AB Tasty is a complete tool for website and conversion rate optimization. We serve as your digital lab, equipped with everything you need to create experiments that will help you to better understand your users and customer journeys so that you can create the clearest and most engaging user experience possible, ensuring your website performs well and yields the maximum results.
Is experimentation for everyone? A resounding yes, says Jonny Longden. All you need are two ingredients: A strong desire and tenacity to implement it.
There’s a dangerous myth lurking around, and it’s the idea that you have to be a large organization to practice experimentation. But it’s actually the smaller companies and start-ups that need experimentation the most, says Jonny Longden of performance marketing agency Journey Further.
With over a decade of experience in conversion optimization and personalization, Jonny co-founded Journey Further to help clients embed experimentation into the heart of what they do. He currently leads the conversion division of the agency, which also focuses on PPC, SEO, PR — among other marketing specializations.
Any company that wants to unearth any sort of discovery should be using experimentation, especially start-ups who are in the explorative phase of their development. “Experimentation requires no size: It’s all about how you approach it,” Jonny shared with AB Tasty’s VP Marketing Marylin Montoya.
Here are a few of our favorite takeaways from our wide-ranging chat with Jonny.
.
The democratization of experimentation
People tend to see more experimentation teams and programs built at large-scale companies, but that doesn’t necessarily mean other companies of different sizes can’t dip their toes in the experimentation pool. Smaller companies and start-ups can equally benefit from this as long as they have the tenacity and capabilities to implement it.
You need to truly believe that without experimentation, your ideas won’t work, says Jonny. There are things that you think are going to work and yet they don’t. Conversely, there are many things that don’t seem like they work but actually end up having a positive impact. The only way to arrive at this conclusion is through experimentation.
Ultimately, the greatest discoveries (for example, space, travel, medicine, etc.) have come from a scientific methodology, which is just observation, hypothesis, testing and refinement. Approach experimentation with this mindset, and it’s anyone’s game.
Building the right roadmaps with product teams
Embedding experimentation into the front of the product development process is important, but yet most people don’t do it, says Jonny. From a pure business perspective, it’s about trying to de-risk development and prove the value of a change or feature before investing any more time, money and bandwidth.
Luckily, the agile methodology employed by many modern teams is similar to experimentation. Both rely on iterative customer collaboration and a cycle of rigorous research, quantitative and qualitative data collection, validation and iteration. The sweet spot is the collection of both quantitative and qualitative data — a good balance of feedback and volume.
The success of building a roadmap for an experimentation program comes down to understanding the organizational structure of a company or industry. In SaaS companies, experimentation is embedded into the product teams; for e-commerce businesses, experimentation fits better into the marketing side. Once you’ve determined the owner and objectives of the experimentation, you’ll need to understand whether you can effectively roll out the testing and have the right processes in place to implement results of a test.
Experimentation is, ultimately, innovation
The more you experiment, the more you drive value. Experimentation at scale enables people to learn and build more tests based on these learnings. Don’t use testing to only identify winners because there’s much more knowledge to be gained from the failed tests. For example, you may only have 1 in 10 tests that work. The real value comes in the 9 lessons you’ve acquired, not just the 1 test that showed positive impact.
When you look at it through these lenses, you’ll realize that the post-test research and subsequent actions are vital: That’s where you’ll start to make more gains toward bigger innovation.
Jonny calls this the snowball effect of experimentation. Experimentation is innovation — when done right. At the root, it’s about exploring and seeing how your customers respond. And as long as you’re learning from the results of your tests, you’ll be able to innovate faster precisely because you are building upon these lessons. That’s how you drive innovation that actually works.
What else can you learn from our conversation with Jonny Longden?
Moving from experimentation to validation
How to maintain creativity during experimentation
Using CRO to identify the right issues to tackle
The required building blocks to successful experimentation
About Jonny Longden
Jonny Longden leads the conversion division of Journey Further, a performance marketing agency specializing in PPC, SEO, PR, etc. Based in the United Kingdom, the part-agency, part-consultancy helps businesses become data-driven and build experimentation into their programs. Prior to that, Jonny dedicated over a decade in conversion optimization, experimentation and personalization, working with Sky, Visa, Nike, O2, Mvideo, Principal Hotels and Nokia.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
Are you on the brink of launching a new feature – one that will affect many of your high-value clients? You’ve worked hard to build it, you’re proud of it and you should be!
You can’t wait to release it for all your users, but wait! What if you’ve missed something? Something that would ruin all your engineering efforts?
There’s nothing worse than starting the day after a release by having to immediately deal with a number of alerts for production issues and spending the day checking a number of logging and monitoring systems for errors and, ultimately, having to rollback the feature you just launched. You would just feel frustrated and unmotivated.
In addition to sapping the morale of your technical teams, NIST has shown that the longer a bug takes to be detected, the more costly it is to fix. This is illustrated by the following graph:
This is explained by the fact that once the feature has been released and is in production, finding bugs is difficult and risky. In addition to preventing users from being affected by problems, it’s critical to ensure service availability.
Are you sure your feature is bug-free?
You might think that this won’t happen to you. That your feature is safe and ready to deploy.
History has shown that it can happen to the biggest companies. Let’s name a few examples.
Facebook, May 7, 2020. An update to Facebook’s SDK rolled out to all users, missed a bug: a server value that was supposed to provide a dictionary of things was changed to provide a simple YES/NO instead. This really tiny change was enough to break Facebook’s authentication system and affect tons of other apps like TikTok, Spotify, Pinterest, Venmo, and other apps that didn’t even use Facebook’s authentication system as it is extremely common for apps to connect to Facebook regardless of whether they use a Facebook-related feature, mainly for ad attribution. The result was unequivocal, the app simply crashed right after launch. Facebook fixed the problem in a hurry, with about two hours for things to get back to normal. But do you have the same resources as Facebook?
Apple, September 19, 2012. Another good example, even though it’s a bit older, would be the replacement of Google Maps with Apple Maps in iOS 6 in 2012 on iOS devices. For many customers and especially fans, Apple always handles the rollout of new features carefully, but this time they messed up. Apple didn’t want to be tied to Google’s app anymore, so they made their own version. However, in their rush to release their map system, Apple made some unforgivable navigational mistakes. Among the many failures of Apple Maps are erased cities, disappearing buildings, flattened landmarks, duplicate islands, distorted graphics, and erroneous location data. A large part of this mess could have been avoided if they had deployed their new map application progressively. They would have been able to spot the bugs and quickly fix them before massive deployment.
And now, thinking about this and seeing that even big companies are impacted, you’re stressed out and may not even want to release it anymore.
But don’t worry! At AB Tasty, we know that building a feature is only half of the story and that to be truly effective, that feature has to be well deployed.
Our feature management service has you covered. You’ll find a set of useful features, such as progressive rollout, to free you from the fear of a release catastrophe and erase feature management frictions, so that you can focus on value-added tasks to get high-quality features into production and apply your energy and innovation in the best way possible, thereby delivering maximum value to your customers.
What’s progressive rollout?
So now you’re curious: what’s progressive rollout? How will this help me monitor the release and make sure everything is okay?
A progressive rollout approach lets you test the waters of a new version with a restricted set of clients. You can set percentages of users to whom your feature will be released and gradually update the percentage to safely deploy your feature. You can also do a canary launch by manually targeting several groups of people at various stages of your rollout.
This is a practice already used by large companies that have realized the significant benefits of a progressive rollout.
Netflix, for example, is one of the most dynamic companies and its developers are constantly releasing updates and new software, but users rarely experience downtime and encounter very few bugs or issues. The company is able to deliver such a smooth experience thanks to sophisticated deployment strategies, such as Canary deployment and progressive deployment, multiple staging environments, blue/green deployments, traffic splitting, and easy rollbacks to help development teams release software changes with confidence that nothing will break.
Disney is another good example of a company that makes the most of progressive deployment. It has taken the phased deployment approach to a whole new level for its “Disney +” and “Star” streaming services by deploying them regionally rather than globally. This delivery method is driven by the needs of the business. The company is making sure that everything is ready at the regional level, in line with its focus on the most important markets. Prior to launching Disney+ in Europe, it spent a lot of time building the local infrastructure needed to deliver a high-quality experience to consumers when launching Disney+ in Europe, including establishing local colocation facilities and beefing up data centers to cache content regionally. After starting to roll it out in Europe, Disney was able to identify that, for some markets, the launch of Disney+ could actually create issues that would have resulted in latency and thus provide a poor experience for affected users. So they took proactive steps to reduce their overall bandwidth usage by at least 25% prior to their march 24 launch and delayed their launch in France by two weeks. Without progressive deployment, they wouldn’t have been able to identify these issues. And that’s why the launching of Disney + was remarkable.
What are the benefits of the progressive rollout?
There are three main benefits to the progressive rollout approach.
Avoiding bugs impacting everyone in production at once
First, by slowly ramping up the load, you can monitor and capture metrics about how the new feature impacts the production environment. If any unanticipated issues come to light, you can pause the full launch, fix the issues, and then smoothly move ahead. This data-driven approach guarantees a release with maximum safety and measurable KPIs.
Validating the “Viable part” in your MVP
You can effectively measure how well your feature is welcomed by your users. If you launch a new feature to 10% of your client base and notice revenue or engagement taking a dip, you can pause the release and investigate. The other major advantage? Anticipating costs. Since margin, profit and revenue are an important part of sustainability, unexpected costs that blow up your projected budgets at the end of the month are almost as bad as the night sweats that come from an unexpected bug! Monitoring your costs during a progressive rollout and immediately pausing the launch if those costs spike is a phenomenal level of control that you will absolutely want to get in on.
Progressively deploying services based upon business drivers
Finally, deploying a service or product progressively can also be seen as a way of prioritizing specific markets based on data-driven business plans. Disney, for example, decided not to launch the service in the U.S. when it launched “Star,” its new channel available in the Disney+ catalog for international audiences, which will feature more mature R-rated movies, FX TV shows, and other shows and movies that Disney owns the rights to but that do not fit the Disney+ family image. Ironically: U.S. customers will have to pay extra on their Disney+ subscription to access the same content on the other streaming service, Hulu.
The decision was made following a complex matrix of rights agreements and revenue streams. Disney found that subscribers are willing to pay for the separate Hulu and Disney+ libraries in the U.S., but that Star’s more limited lineup was enough to justify a standalone paid purchase for international customers, who will have to add $2 to their initial $6.99 subscription to access it. When the content library for Star is enough to justify not going through Hulu anymore, the U.S. customers will have access to it by paying just 1$ more. This progressive rollout approach has enabled Disney to make sure that once they launch Star in the U.S., everything will be ready and they will achieve good results.
In other words, the progressive rollout approach helps you ensure that your functionality meets the criteria of usability, viability, and desirability in accordance with your business plan.
How to act fast when you identify bugs while progressively deploying a feature?
Now that you know more about the progressive rollout of your features/products, you may be wondering how to take action if you identify bugs or if things aren’t going well. Lucky for you, we’ve thought of that part too. In addition to progressive rollout, you’ll also find automatic rollback on KPIs and feature flagging in the AB Tasty toolkit.
Feature flagging will let you set up flags on your feature, that work as simply as a switch on/off button. If for any reason you identify threats in your rollout or if the engagement of your users is not really convincing, you can simply toggle your feature off and take time to fix any issues.
This implies that you are aware and that someone from the product team is available to turn it off. But what if something happens overnight and no one can check on the progress of the deployment? Well, for that eventuality, you can set up automatic rollbacks (also called Rollback Threshold) linked to key performance indicators. Our algorithm will check the performance of your deployment and, based on the KPIs you set, if something goes wrong, it will automatically roll back the deployment and inform you that a problem has occurred. This way, in the morning, your engineers will be able to fix the problems without having to deal with the rollback themselves.
Conclusion
Downtime incidents are stressful for both you and your customers. To resolve them quickly and efficiently, you need to have access to the right tools and make the most of them. The progressive rollout, automatic rollback, and feature flagging are great levers to relieve your product teams of stress and let them focus on innovating your product to create a wonderful experience for your users. Highly effective organizations have already realized the importance of having the right approach to deployment with the right tools. What about your organization?
AB Tasty minimizes risk and maximizes results to make the lives of Product teams a whole lot easier. Create a free account today!
Chad Sanderson breaks down the most successful types of experimentations based on company size and growth ambitions
For Chad Sanderson, head of product – data platform at Convoy, the role of data and experimentation are inextricably intertwined.
At Convoy, he oversees the end-to-end data platform team — which includes data engineering, machine learning, experimentation, data pipeline — among a multitude of other teams who are all in service of helping thousands of carriers ship freight more efficiently. The role has given him a broad overview of the process, from ideation, construction to execution.
As a result, Chad has had a front-row seat that most practitioners never do: The end-to-end process of experimentation from hypothesis, data definitions, analysis, reporting to year-end financials. Naturally, he had a few thoughts to share with AB Tasty’s VP Marketing Marylin Montoya in their conversation on the experimentation discipline and the complexities of identifying trustworthy metrics.
Introducing experimentation as a discipline
Experimentation, despite all of its accolades, is still relatively new. You’ll be hard pressed to find great collections of literature or an academic approach (although Ronny Kohavi has penned some thoughts on the subject matter). Furthermore, experimentation has not been considered a data science discipline, especially when compared to areas of machine learning or data warehousing.
While there are a few tips here and there available from blogs, you end up missing out on the deep technical knowledge and best practices of setting up a platform, building a metrics library and selecting the right metrics in a systematic way.
Chad attributes experimentation’s accessibility as a double-edged sword. A lot of companies have yet to apply the same rigor that they do to other data science-related fields because it’s easy to start from a marketing standpoint. But as the business grows, so does the maturity and the complexity of experimentation. That’s when the literature on platform creation and scaling is scant, leading to the field being undervalued and hard to recruit the right profiles.
When small-scale experimentation is your best bet
When you’re a massive-scale company — such as Microsoft or Google with different business units, data sources, technologies and operations — rolling out new features or changes is an incredibly risky endeavour, considering that fact that any mistake could impact millions of users. Imagine accidentally introducing a bug for Microsoft Word or PowerPoint: The impact on the bottom line would be detrimental.
The best way for these companies to experiment is with a cautious, small-scale approach. The aim is to focus on immediate action, catching things quickly in real time and rolling them back.
On the other hand, if you’re a startup in a hyper-growth stage, your approach will vastly differ. These smaller businesses typically have to show double-digit gains with every new feature rollout to their investors, meaning their actions are more so focused on proving the feature’s positive impact and the longevity of its success.
Make metrics your trustworthy allies
Every business will have very different metrics depending on what they’re looking for; it’s essential to define what you want before going down the path of experimentation and building your program.
One question you’ll need to ask yourself is: What do my decision-makers care about? What is leadership looking to achieve? This is the key to defining the right set of metrics that actually moves your business in the right direction. Chad recommends doing this by distinguishing your front-end and back-end metrics: the former is readily available, the latter not so much. Client-side metrics, what he refers to as front-end metrics, measure revenue per transaction. All metrics then lead to revenue, which in and of itself is not necessarily a bad thing, but that just means all your decisions are based on revenue growth and less on proving the scalability or winning impact of a feature.
Chad’s advice is to start with the measurement problems that you have, and from there, build out your experimentation culture, build out the system and lastly choose a platform.
What else can you learn from our conversation with Chad Sanderson?
Different experimentation needs for engineering and marketing
Building a culture of experimentation from top-down
The downside of scaling MVPs
Why marketers are flagbearers of experimentation
About Chad Sanderson
Chad Sanderson is an expert on digital experimentation and analysis at scale. He is a product manager, writer and public speaker, who has given lectures on topics such as advanced experimentation analysis, the statistics of digital experimentation, small-scale experimentation for small businesses and more. He previously worked as senior program manager for Microsoft’s AI platform. Prior to that, Chad worked for Subway’s experimentation team as a personalization manager.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
One of the pioneers of experimentation shares a humbling reality check: Most ideas will fail (and it’s a good thing)
Few people have accumulated as much experience as Ronny Kohavi when it comes to experimentation. His work at tech giants such as Amazon, Microsoft and Airbnb — just to name a few — has laid the foundation of modern online experimentation.
Before the idea of “build fast, deploy often” took hold across tech companies, developers followed a waterfall model that saw fewer releases (sometimes every 2-3 years). The shortening of development cycles in the early 2000s thanks to the Agile methodology and an uptick in online experimentation created the perfect storm for a software development revolution ― and Ronny was at the center of it all.
AB Tasty’s VP Marketing Marylin Montoya set out to uncover the early days of experimentation with Ronny and why failure is actually a good thing. Here are some of the key takeaways from their conversation.
.
Progressive deployments as a safety net
A typical cycle of experimentation involves exposing the test to 50% of the population for an average of two weeks before a gradual release. But Ronny suggests coming at it from a different vantage point: Starting with a small audience of just 2% before ramping up to 50%. The slower ramp-up gives you the time to detect any egregious issues or a degradation in metric values in near real time.
In an experiment, we may focus on just two features, but we have a large set of guardrails that suggest we shouldn’t be degrading X, Y or Z. Statistical data that you’re collecting could also suggest that you’re impacting something you didn’t mean to. Hence, the usage of progressive deployments in which you can identify external factors and easily rollback your test.
It’s like if you’re cooling water: You may realize you’re changing the temperature, but it’s not until you reach 0ºC (32ºF) that ice forms. You suddenly realize that when you get to a certain point, something very big happens. So, deploying at a safe velocity and monitoring the results can lead to huge improvements.
Your great idea? It will most likely fail.
Nothing gives you a better reality check than experimentation at scale. Everyone thinks they’re doing the best stuff in the world until it’s in the hands of their users. That’s when the real feedback kicks in.
Over two-thirds of ideas actually fail to move the metrics that they were designed to improve — a statistic Ronny shares from his time at Microsoft, where he founded the experimentation platform team of more than 100 data scientists, developers and program managers.
Don’t be deterred, however. In the world of experimentation, failing is a good thing. Fail fast, pivot fast. Being able to realize that the direction you’re going in isn’t as promising as previously thought enables you to use those new findings to enrich your next actions.
At Airbnb, Ronny’s experimentation team deployed a lot of machine learning algorithms to improve search. Out of 250 ideas tested in controlled experiments, only 20 of them proved to have a positive impact on the key metrics — meaning over 90% of ideas failed to move the needle. On the flip side, however, the 20 ideas that did succeed in some form? Those resulted in a 6% improvement in booking conversion, worth hundreds of millions of dollars.
The starter kit to experimentation
It’s easier today to convince leadership to invest in experimentation because there are plenty of successful use cases out there. Ronny’s advice is to start with a team that has iteration capital. If you’re able to run more experiments and a certain percentage are pass/fail, this ability to try ideas is key.
Pick a scenario where you can easily integrate the experimentation process into the development cycle and then work your way on to more complex scenarios. The value of experimentation is clearer because deployments are happening more often. If you’re working in a team that deploys every six months, there’s not a lot of wiggle room because everyone has already invested their efforts into this idea that the feature cannot fail. Which, as Ronny pointed out earlier, has a low probability of success.
Is experimentation for every company? The short answer is no. A company has to have certain ingredients in order to unlock the value of experimentation. One ingredient you need is being in a domain where it’s easy to make changes, such as website services or software. A second ingredient is you need enough users. Once you have tens of thousands of users, you can start experimenting and doing it at scale. And lastly, make sure you have trustworthy results from which you are taking your decisions.
What else can you learn from our conversation with Ronny Kohavi?
How experimentation becomes central to your product build
Why experimentation is at the root of top tech companies
The role leaders play in evangelizing an experimentation culture
How to build an environment for true experimentation and trustworthy results
About Ronny Kohavi
Ronny Kohavi is an authority in experimentation, having worked on controlled experiments, machine learning, search, personalization and AI for nearly three decades. Ronny previously was vice president and technical fellow at Airbnb. Prior to that, Ronny led the Analysis and Experimentation at Microsoft’s Cloud and AI group and was the director of data mining and personalization at Amazon. Ronny has also co-authored “Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing.,” which is currently the #1 best-selling data-mining book on Amazon.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
Statistical significance is a powerful yet often underutilized digital marketing tool.
A concept that is theoretical and practical in equal measures, you can use statistical significance models to optimize many of your business’s core marketing activities (A/B testing included).
A/B testing is integral to improving the user experience (UX) of a consumer-facing touchpoint (a landing page, checkout process, mobile application, etc.) and increasing its performance while encouraging conversions.
By creating two versions of a particular marketing asset, both with slightly different functions or elements, and analyzing their performance, it’s possible to develop an optimized landing page, email, web app, etc. that yields the best results. This methodology is also referred to as two-sample hypothesis testing.
When it comes to success in A/B testing, statistical significance plays an important role. In this article, we will explore the concept in more detail and consider how statistical significance can enhance the A/B testing process.
But before we do that, let’s look at the meaning of statistical significance.
What is statistical significance and why does it matter?
According to Investopedia, statistical significance is defined as:
“The claim that a result from data generated by testing or experimentation is not likely to occur randomly or by chance but is instead likely to be attributable to a specific cause.”
In that sense, statistical significance will bestow you with the tools to drill down into a specific cause, thereby making informed decisions that are likely to benefit the business. In essence, it’s the opposite of shooting in the dark.
Make informed decisions with testing and experimentation
Calculating statistical significance
To calculate statistical significance accurately, most people use Pearson’s chi-squared test or distribution.
Invented by Karl Pearson, the chi (which represents ‘x’ in Greek)-squared test commands that users square their data to highlight possible variables.
This methodology is based on whole numbers. For instance, chi-squared is often used to test marketing conversions—a clear-cut scenario where users either take the desired action or they don’t.
In a digital marketing context, people apply Pearson’s chi-squared method using the following formula:
Statistically significant = Probability (p) < Threshold (ɑ)
Based on this notion, a test or experiment is viewed as statistically significant if the probability (p) turns out lower than the appointed threshold (a), also referred to as the alpha. In plainer terms, a test will prove statistically significant if there is a low probability that a result has happened by chance.
Statistical significance is important because applying it to your marketing efforts will give you confidence that the adjustments you make to a campaign, website, or application will have a positive impact on engagement, conversion rates, and other key metrics.
Essentially, statistically significant results aren’t based on chance and depend on two primary variables: sample size and effect size.
Statistical significance and digital marketing
At this point, it’s likely that you have a grasp of the role that statistical significance plays in digital marketing.
Without validating your data or giving your discoveries credibility, you will probably have to take promotional actions that offer very little value or return on investment (ROI), particularly when it comes to A/B testing.
Despite the wealth of data available in the digital age, many marketers are still making decisions based on their gut.
While the shooting in the dim light approach may yield positive results on occasion, to create campaigns or assets that resonate with your audience on a meaningful level, making intelligent decisions based on watertight insights is crucial.
That said, when conducting tests or experiments based on key elements of your digital marketing activities, taking a methodical approach will ensure that every move you make offers genuine value, and statistical significance will help you do so.
Using statistical significance for A/B testing
Now we move on to A/B testing, or more specifically, how you can use statistical significance techniques to enhance your A/B testing efforts.
Testing uses
Before we consider its practical applications, let’s consider what A/B tests you can run using statistical significance:
Emails clicks, open rates, and engagements
Landing page conversion rates
Notification responses
Push notification conversions
Customer reactions and browsing behaviors
Product launch reactions
Website calls to action (CTAs)
The statistical steps
To conduct successful A/B tests using statistical significance (the chi-squared test), you should follow these definitive steps:
1. Set a null hypothesis
The idea of the null hypothesis is that it won’t return any significant results. For example, a null hypothesis might be that there is no affirmative evidence to suggest that your audience prefers your new checkout journey to the original checkout journey. Such a hypothesis or statement will be used as an anchor or a benchmark.
2. Create an alternative theory or hypothesis
Once you’ve set your null hypothesis, you should create an alternative theory, one that you’re looking to prove, definitively. In this context, the alternative statement could be: our audience does favor our new checkout journey.
3. Set your testing threshold
With your hypotheses in place, you should set a percentage threshold (the (a) or alpha) that will dictate the validity of your theory. The lower you set the threshold—or (a)—the stricter the test will be. If your test is based on a wider asset such as an entire landing page, then you might set a higher threshold than if you’re analyzing a very specific metric or element like a CTA button, for instance.
For conclusive results, it’s imperative to set your threshold prior to running your A/B test or experiment.
4. Run your A/B test
With your theories and threshold in place, it’s time to run the A/B test. In this example, you would run two versions (A and B) of your checkout journey and document the results.
Here you might compare cart abandonment and conversion rates to see which version has performed better. If checkout journey B (the newer version) has outperformed the original (version A), then your alternative theory or hypothesis will be proved correct.
5. Apply the chi-squared method
Armed with your discoveries, you will be able to apply the chi-squared test to determine whether the actual results differ from the expected results.
To help you apply chi-squared calculations to your A/B test results, here’s a video tutorial for your reference:
By applying chi-squared calculations to your results, you will be able to determine if the outcome is statistically significant (if your (p) value is lower than your (a) value), thereby gaining confidence in your decisions, activities, or initiatives.
6. Put theory into action
If you’ve arrived at a statistically significant result, then you should feel confident transforming theory into practice.
In this particular example, if our checkout journey theory shows a statistically significant relationship, then you would make the informed decision to launch the new version (version B) to your entire consumer base or population, rather than certain segments of your audience.
If your results are not labelled as statistically significant, then you would run another A/B test using a bigger sample.
At first, running statistical significance experiments can prove challenging, but there are free online calculation tools that can help to simplify your efforts.
Statistical significance and A/B testing: what to avoid
While it’s important to understand how to apply statistical significance to your A/B tests effectively, knowing what to avoid is equally vital.
Here is a rundown of common A/B testing mistakes to ensure that you run your experiments and calculations successfully:
Unnecessary usage: If your marketing initiatives or activities are low cost or reversible, then you needn’t apply strategic significance to your A/B tests as this will ultimately cost you time. If you’re testing something irreversible or which requires a definitive answer, then you should apply chi-squared testing.
Lack of adjustments or comparisons: When applying statistical significance to A/B testing, you should allow for multiple variations or multiple comparisons. Failing to do so will either throw off or narrow your results, rendering them unusable in some instances.
Creating biases: When conducting A/B tests of this type, it’s common to apply biases to your experiments unwittingly—the kind of which that don’t consider the population or consumer base as a whole.
To avoid doing this, you must examine your test with a fine-tooth comb before launch to ensure that there aren’t any variables that could push or pull your results in the wrong direction. For example, is your test skewed towards a specific geographical region or narrow user demographic? If so, it might be time to make adjustments.
Statistical significance plays a pivotal role in A/B testing and, if handled correctly, will offer a level of insight that can help catalyze business success across industries.
While you shouldn’t rely on statistical significance for insight or validation, it’s certainly a tool that you should have in your digital marketing toolkit.
We hope that this guide has given you all you need to get started with statistical significance. If you have any wisdom to share, please do so by leaving a comment.
Any business selling products or services online has a conversion funnel — but not everyone realizes it. If you’re unsure what a conversion is or how you can refine yours to sell more online, you’re in the right place. In this post, we’re going to take you through everything you need to know about conversion funnels. We’ll start with the basics — what conversion funnels are and the three key stages — before moving on to some of the most effective strategies to improve your funnels to increase sales. Let’s get stuck in!
In this article, we’ll cover:
[toc]
What is a conversion funnel?
A conversion funnel is a process that takes potential customers on a journey towards buying your products or services. They’re the cornerstone of all e-commerce business models, guiding potential customers from the moment they first become aware of your brand to the moment they make a purchase and beyond.
If you’re new to conversion funnels, think about the shape of a funnel — it’s wider at the top and narrower at the bottom. This represents the flow of people through your marketing strategy. Not everyone who becomes aware of your business will go on to become a paying customer. It’s like brewing coffee using a drip filter — a large volume of coffee grounds go into the top of the brewing equipment and then the funnel filters the high-quality stuff out of the bottom into your mug. A sales funnel works in the same way. The goal is to get as many relevant leads into the top of the funnel as possible, filtering out unsuitable prospects to leave your ideal customers ready to buy from you.
When you optimize your conversion funnel, you maximize the impact of your online marketing strategy and boost sales. This isn’t a once-and-done exercise, but something you need to continually refine throughout your business life. Do you want to know how to do it?
What’s the difference between a conversion funnel and a sales funnel?
The terms conversion funnel and sales funnel are often used interchangeably, but are they the same thing? The answer to this question is no, although they are closely related. A sales funnel typically starts when a potential customer enters the sales pipeline. This can happen online (in an e-commerce environment) as well as offline. However, a prospect typically doesn’t enter your sales funnel until they’re already familiar with your brand and your products or services.
It can take a while to get to this point in the online world, particularly if you’re targeting people who have never heard of your brand before. It takes time to build a connection and trust with your audience.
This is where a conversion funnel comes in. Here, the focus isn’t just on making a sale. It’s about making a connection with your audience, generating leads, and then taking those leads on a journey with your company. Potential customers might come into your funnel cold, without much awareness of who you are or what you do. Over time, your funnel will warm them up, build trust in your offer, and get them ready to buy. It encapsulates the whole process — from the first contact through to purchasing.
The three conversion funnel stages
There are many different conversion funnel models out there. All of them broadly suggest the same thing: breaking the process down into several conversion funnel stages the leads must travel through before making a purchase. Although a customer may enter or exit the funnel at any stage, your personalized model sets out how you intend customers to connect with your business.
The exact model will look different for every organization, but here are the three stages we suggest you follow.
Stage 1: Building awareness at the top of the funnel
The top of the funnel is all about making people aware of your brand and capturing leads. This stage is arguably the most crucial. If you don’t get people into your funnel, how are you going to sell to them? This critical step is often referred to as the awareness stage, and the exact strategy you use to do this will depend on your ideal customer. Who are they? Where do they hang out? What are their fundamental problems and challenges? Why would they be interested in what you have to offer them? The answers to these questions can provide useful directions during the awareness stage. Remember: this isn’t about you; it’s about the customer. Here are a few things that should be happening at the top of the funnel.
Content marketing
To grab attention online, you’re going to need content. This content can take many forms, so it’s essential to think about the types of content your audience is most likely to consume. For example, TikTok videos will likely appeal to 18 to 24-year-olds, but they might not be the best option if you’re targeting an older demographic.
You should consider both onsite and offsite content when outlining your content marketing strategy. An effective conversion funnel needs both. Offsite content helps capture attention and attract people to your website. In contrast, onsite content engages your audience and encourages them to take the next step, such as signing up for your mailing list.
Marketing campaigns
Alongside your content marketing strategy, you should also consider the marketing campaigns you will be running to get people to engage with this content. How will you get your content seen? How will you capture users’ attention? Are you only operating online, or will you use offline marketing to generate leads?
Often, e-commerce businesses are quick to dismiss offline marketing campaigns as irrelevant. However, highly targeted offline campaigns can be extremely useful. The online marketplace is crowded! If you can think of innovative ways to reach your audience offline and direct them to your online content, it could turn out to be a cost-effective way to generate leads for your conversion funnel.
You could also consider how you might automate some of your marketing campaigns. Creating evergreen campaigns that can run in the background while you and your employees focus on other tasks is useful to maximize profits. In essence, it means you can be generating leads for your business while you sleep.
Lead capture
Lead capture is the final step of the awareness stage. It’s where you move your prospects from the top of your conversion funnel to the middle. Once you’ve directed a potential customer to your website and encouraged them to engage with your content, what’s next? Each piece of content your audience engages with on your website should have a call to action — something that tells them what action to complete next.
To achieve this, you might want to consider a lead magnet. This can be something as simple as a discount code. But, for maximum results, you could develop something that helps solve a problem directly related to the product or service you’re offering.
Not only does this ensure you’re capturing highly qualified leads, but it also means people are likely to sign up even when they’re not ready to make a purchase. Given the point of a conversion funnel is to get them ready to buy from you, this is a vital point to consider when outlining your content marketing strategy.
Once you have that email address, it’s time to move on to the second stage of the conversion funnel: nurturing your audience to build desire for your products or services.
To maximize the number of leads you’re capturing, you should focus your stage one activities across a range of digital marketing channels. Here are some of the most popular options:
Social media
Given there are almost 4 billion social media users worldwide (over half the world’s population), it’s no surprise social media marketing is one of the most popular ways to generate leads. That said, it’s important to note it isn’t an easy option! Many business owners expect social media to be a fast and cheap way to grow an audience. Still, it takes time and persistent effort to get results — just like any other marketing strategy.
Work with a professional to develop a social media marketing plan that helps you stand out from the crowd. Many businesses use social media to attract people into their conversion funnel, but few do it well.
Paid search
What’s the first place you turn to when you need information? It’s estimated there are around 2 trillion Google searches every year — so advertising your content on Google could potentially be very lucrative! Unlike social media marketing, people using search engines are actively looking for the information you’re providing. To get the best click-through rate, make sure the phrases you’re targeting are directly relevant to the content. And test campaigns with a small budget before increasing your spending.
Organic search
It’s also a good idea to optimize your content for organic search. While this isn’t a short-term strategy, Search Engine Optimization (SEO) can deliver large volumes of traffic to your website over time. Focus on creating evergreen content — content that doesn’t become irrelevant or outdated and can appear in organic searches for many years to come. When you gain website visitors organically from search engines, you improve your ability to build a list of qualified leads, improving the quality of people entering your conversion funnel.
Stage 2: Nurturing your audience
Many online businesses make the fundamental mistake of pushing for a sale too soon. While you can (and should) always have an option for potential customers to buy from you on their terms, you should design your conversion funnel to nurture your leads, building trust with your brand before moving them into the sales pipeline.
Staying in contact
Once a potential customer has told you they want to hear more from you, it’s essential to stay in touch with them. If you can, you should aim to use multiple channels to do this. Encourage them to follow you on social media, re-target them with relevant online content, and send them regular emails. Research consistently shows the more opportunities a potential customer has to engage with your brand online, the more likely they will buy from you.
In short, it’s not enough to let people know you exist. If you want to sell to them, you need to put in the work to keep them engaged!
Positioning your products and services
As you stay in touch and nurture your audience, you should also ensure each lead is familiar with your products and services. This step isn’t about pushing for the sale — we’ll come back to this in the next stage — but you should be introducing your offering interestingly and engagingly. Essentially, we need your leads to be ready to make a purchase when you deliver your sales pitch. To get to this stage, they need to know what you’re selling.
Building a desire to buy
And finally, throughout the nurturing stage, you should be gearing up your audience to perform the desired action. In most cases, this is completing a purchase. How do you do this? Use emotion.
Humans are emotional beings. Remember earlier when we discussed the problems and challenges your product or service can solve for your customers? What are the emotions behind that problem? Aim to appeal to these emotions when engaging with your audience, and make it clear that you’re here to help them overcome these feelings to foster more positive and desirable emotions. How will your product or service make them feel? Can you impart some of these feelings with your content?
As well as feeling emotion, people have an inbuilt desire to be understood. The more you can show them you understand them, the more they will connect with your brand, and the more desire they will have to do business with you.
Throughout this step, you should be keeping your competitors in mind, especially if you’re operating in a competitive niche. Why should your audience choose you above your competition?
Stage 3: Convert potential customers into paying customers
Stage three is what it’s all about — securing the sale. Without this stage, your business is nothing — without paying customers, you have no profits. But we hope you now appreciate why it’s important to take your audience on a journey through the preceding stages before you attempt to convert them. Once you’ve optimized your funnel, your leads will now be ready to buy from you.
Continue to nurture leads
It’s crucial to be aware of this: you don’t stop nurturing your prospects once you get them to the end of your funnel. This stage should continue as long as your leads — and eventual customers — are in contact with your business.
Work at your potential customer’s pace
It’s also important to remember your potential customers will all travel at their own pace. Some will be ready to make a purchase sooner than others. For this reason, you should think of your conversion funnel as a process. It isn’t about throwing leads in at one end and spitting them out at the other side but about fostering connections that will help your organization thrive over time.
If you attempt to trigger a sale, but your customers aren’t ready, you should continue to engage and nurture them — and try again further down the line. Similarly, if none of your prospects are buying from you at this stage in your conversion funnel, it’s a sign something needs tweaking — we’ll get back to this in a little while.
Trigger a Sale
Now it’s time to encourage your leads to become paying customers, but how should you do it? As always, there are many options here. Finding the right approach will likely involve some trial-and-error. It’s a good idea to test out a few sales tactics and see what works. For some, a simple email or retargeting campaign on social media might do the trick. But for other businesses, you might need to come up with something more personal or creative.
What makes a good call-to-action?
Calls-to-action are the lifeblood of any effective conversion funnel. But how can you make sure yours are effective? Here are some tips to get you started.
Be clear and concise
Your call-to-action shouldn’t be too wordy. It would be best if you were direct. Use short sentences and tell your audience exactly what you want them to do. Use verbs like “buy,” “shop,” or “download.” Telling someone to “shop the new collection” is likely to result in more sales than something like “our new collection is now live on our website.”
Ask yourself why
As you develop your call-to-action, put yourself in your potential customer’s shoes. Why should they do what you’re asking them to? This is where the copy in the rest of your sales pitch comes in. The call-to-action is the final piece of the puzzle. By the time your lead gets to this part of your content, they should already be ready to hit that button. Make it a no-brainer for them.
The role of the shopping cart
The shopping cart on your website can be one of your biggest assets for driving sales. Did you know you can follow up on abandoned carts with your email subscribers? If not, you’re missing out on one of the most effective conversion tools available to e-commerce businesses. Research suggests around 70% of all shopping carts are abandoned online. Think about it: these are leads that have been through the conversion funnel and are almost ready to make a purchase. What is it that stopped them? It might have been something as simple as an interruption. Get back in touch and ask them if they’re ready to complete their purchase. The results may surprise you.
Evaluating your funnel with conversion funnel metrics
As we mentioned at the start of this post, a conversion funnel isn’t something you can create and then forget about. It’s an ongoing, interactive process that you must refine over time. The digital marketing world is dynamic and ever-changing — and your conversion funnel will need to evolve alongside industry trends and technological advances. Evaluating your funnel is an essential part of this, enabling you to improve each stage of the process to generate more qualified leads and convert more of them into paying customers.
Your first step should be to set up Google Analytics to track your conversion funnel. When you do this, you can track a lead from the moment they join your funnel until they make a purchase. This gives you an overview of how well your funnel is performing, as well as helping you access some of the key conversion funnel metrics that help you decide what to focus on next, such as:
Cost per acquisition (CPA)
Marketing costs money and the expenses associated with your conversion funnel can quickly mount up. It’s vital to understand the benefit these investments bring. What is the return on investment (ROI) associated with your conversion funnel? To understand this, you need to calculate your cost per acquisition. To calculate this, divide the costs associated with your conversion funnel by the number of paying customers the funnel generated in the same time period. For example, if you invested $500 and generated 10 paying customers, your CPA would be $50.
You can then compare this with the average spend to figure out whether your conversion is profitable or not. Using the example above, if the average customer spends $200, your funnel is profitable. On the other hand, if the average lifetime spend is $20, the funnel is operating at a loss.
Conversion rate
Google Analytics calculates your funnel’s conversion rate by working out how many of the visitors went to the goal page (e.g., “thank you for your purchase”) as well as one of the pages associated with the earlier stages of your conversion funnel. This provides you with useful insight into how well your funnel is working over time, which can help you evaluate any changes that you make to optimize the funnel.
Are you ready to optimize your funnel?
In summary, conversion funnels are an essential asset to all e-commerce businesses. If you want to improve sales, optimizing your funnel is often the best place to start. What steps will you take after reading this post?
You may not be aware of this, but it’s likely that you’ve come across the serial position effect on more than one occasion.
A concept coined by renowned psychologist, Hermann Ebbinghaus, the serial position effect refers to how the location of an item in a sequence influences a person’s memory or recall.
The concept dictates that people usually remember items at the beginning or the end of a list or sequence with greater accuracy than those in the middle.
User experience (UX) designers leverage the serial position effect to improve their designs and create a richer, more seamless experience for consumers. This approach to digital design is present in the websites, apps or landing pages of iconic brands such as Apple, Nike or Electronic Arts (EA).
Here we’re going to explore the serial position effect in more detail, explore some notable design examples, and consider how you can use this powerful principle to improve your brand’s UX offerings.
What is the serial position effect?
When it comes to UX optimization, the order of things matter. As humans, we do indeed tend to remember the items near the start or end of a list — much like our brains respond well to storytelling.
Hermann Ebbinghaus coined the phrase based on in-depth studies on the short as well as long term memory and its impact on how we remember or perceive information. These studies were further developed by psychologists B.Murdock in 1962 and Glanzer & Cunitz in 1966.
These extensive studies resulted in the two vital serial position effect concepts: the primacy effect and the recency effect.
Primacy effect
The primacy effect is based on the discovery that an individual is likely to recall items, assets or information from the start of a list.
For instance, when someone attempts to remember something from a long list of words, they are likely to recall the terms words listed at the beginning, rather the middle.
As such, the primacy effect helps a user to remember the information they absorb first better than the information they see later on in their journey (further down a landing page, for example).
Recency effect
Essentially, the recency effect is a concept contrary to the primary effect. Rather than recalling information absorbed earlier on, the recency effect is based on the notion of people remembering the information they see last with more clarity. This model is dependent on short-term memory.
A mix of studies suggests that the recency effect is prevalent in thecourtroom. In many cases, jurors are more likely to recall, and agree with, the argument or conclusion they hear last.
In a UX design context, for instance, a potential customer will recall the last two items they saw on a personalized product recommendation carousel and purchase one of these products as a result.
The primacy and recency effect combined make up key elements of the serial position effect, which brings us onto our next point.
Applying the serial position effect to design
Now that you understand the fundamental concepts of the serial position effect, we’re going to consider how you can apply it to design — or more specifically, to user design interfaces.
Both the primacy and recency effect can have a significant impact on the design of user interfaces. Extensive lists of information put a strain on the human memory, often hindering perception and recall; and, by utilizing both ends of the serial position effect spectrum (primacy and recency), you can enhance your designs significantly.
By understanding that items or assets in the middle of a sequence are usually absorbed the least, it’s possible to leverage the serial position effect to minimize the loss of information. In doing so, it’s possible to create interface designs that are richer, more valuable, and easier to navigate.
Considering that 38% of consumers will bounce off a web page if its layout is poor or unattractive, getting your design right will prove critical to your long term success.
Applying the serial position effect to your interface design process is at its core, down to ensuring that users can navigate the items or information on your page intuitively.
If your design is digestible, fluid, and seamless, users will recall vital information with more clarity while taking desired actions like signing up to a newsletter or buying a specific product.
Here are four essential principles of applying the serial position effect to interface design:
1. Provide practical, task-relevant information
Adding and maintaining task-relevant information to your interface will not only make your design more engaging, but it will reduce the strain on users’ focus or recall.
Publishing platform Medium, for instance, has designed its user interface to simplify its interactions from a reader’s as well as a writer’s perspective.
With a host of visual tools tailored to the users’ preferred topics or interests, you gain a visual snapshot of information that offers access to relevant content and to your reading list, and allows you to create a new piece of content with swift, seamless actions.
2. Add recognizable cues
Adding dynamic cues to your user interface design minimizes cognitive strain while facilitating informational recall.
Audible notifications (e.g. pings when you receive a message) or textual cues (e.g. small informational pop-up boxes) create a real sense of recognition. Video games like ‘Need For Speed’ or ‘Broken Sword’ are excellent examples of cue-based design for user interfaces.
EA Games’ once popular ‘Plants vs Zombies’ game, for instance, utilizes a multitude of recognizable visual and audio cues to help players navigate their way through the game and remain ‘in the moment’ without pushing them to their cognitive limits.
Foley-style sounds unique to each move the player makes (planting sounds, digging sounds etc.), text-based captions that tell the player what to expect next, and visual icons at the top of the screen all work cohesively to make the user experience feel as natural as possible. You can apply similar cues to e-commerce sites to enrich your designs and make them more intuitive.
3. Reduce the level of recall required
The human attention span has its limits and, typically, can only retain five pieces of information at any one time.
If you prioritize limiting the necessity for recall, you will guide users through their journey in a way that helps them remember relevant information as and when required.
Technology colossus Apple utilizes a visual grid system with informational titles and scannable dropdown boxes to help its customers comprare models with ease and pick a product that suits their specific needs. At any one point in the interface journey, users are only presented with the information they need — details including essential specs, main comparisons, and price.
This simple yet effective design prioritizes the most valuable information, minimizing the need for recall in the process.
4. Emphasize essential information at the start and end
Playing directly into the hands of the primacy and recency effect, highlighting or placing the most essential information at the start and the end (or the top and bottom) of your interface, placing the less important items in the center.
World-renowned e-commerce leader Amazon, for example, displays digestible personalized prompts, commands, and information at the top of its homepage.
In the center of the page, you gain access to trending products and deals. At the bottom of the page, or interface, you’re presented with personalized suggestions based on your shopping history or browsing behavior:
This design technique maximizes the potential for users to recall the information that offers the most value or is likely to prompt further engagement. An effective approach that enriches the user experience while increasing the chances of regular consumer conversions.
“Design used to be the seasoning you’d sprinkle on for taste; now it’s the flour you need at the start of the recipe.”
— John Maeda, design & UX expert
Serial position effect for landing page UX
From the user interface design methods we’ve explored, it’s clear that the order, as well as the way you present information, have a significant impact on how people interact with your brand or business.
In today’s hyper-connected digital age, your UX offerings count more than ever. 88% of users are unlikely to return to a website or landing page after a poor user experience.
To enhance your landing page UX and create an experience that will increase engagement while encouraging customer loyalty, you should consider implementing the serial position effect.
To reiterate the impact the serial position effect can have on landing page UX, here’s a visualization of the serial position curve.
From a digital marketing perspective, the serial position curve clearly demonstrates that people recall information towards the start and end of an informational sequence, with items or messaging in the middle of a landing page absorbed least. It’s a steady consistent curve that can offer a practical framework for your landing pages’ UX designs.
Russian e-commerce brand, Marc Cony, uses the serial effect methodology to increase new user engagement through its primary landing page.
Marc Cony homepage highlighting discount information(Source)
Here, you can see that the landing page design is clean and minimal to simplify user navigation while highlighting its most engagement-driving messaging as soon as you visit.
As you navigate your way down the landing page, there is a clear hierarchy of information. Scroll down and you’re presented with the opportunity to personalize your shopping experience, before viewing content surrounding the brand’s blog and social media pages.
Finally, there is a clean, concise call to action (CTA) button that prompts you to sign up to the brand’s newsletter and ‘convert.’ This is an excellent example of how using serial effect principles can create a seamless user experience while guiding consumers towards a desired action — in this case, viewing sale items or becoming an email subscriber.
Online retail innovator, Thread, offers an interactive and visually-rich approach to reduce consumer recall and optimize its landing page for increased brand engagement.
Thread’s clean, grid-based design is easy to scan and it’s above the fold messaging prompts the user to take action without having to second-guess themselves.
Thread homepage visually-rich approach
This interactive approach offers personal value while offering an incentive to interact. Clicking on preferred styles requires minimal recall and, as such, keeps the information at the top of the page fresh in the mind of the consumer.
Thread website, subheadings navigation
Once you’ve selected your preferred styles, you’re directed to a new landing page. Clear subheadings help you navigate your way through the page with minimal cognitive strain, and once you reach the bottom, the ‘Next’ CTA tells you what to do.
This approach to the serial position effect helps to streamline the user experience while keeping consumers engaged in the brand at all times.
A well-crafted informational hierarchy and interactive visual approach is a testament to the power of presenting information effectively without overwhelming the user with unnecessary data. This is definitely a driving force behind the startup’s ongoing success!
Whether you’re selling goods or services, applying the serial position effect will help you improve your landing pages’ UX and increase your conversion rates.
The Digital Marketing Institute, primacy and recency effect on Homepage (Source)
Digital marketing course provider, the Digital Marketing Institute, utilizes both the primacy and recency effect to UX optimize many of its landing pages.
The DMI’s homepage, for example, includes a clearly labelled ‘Download Brochure’ button at the very top of the page. The main banner tells the user exactly what the brand does and how they will benefit from enrolling (using a second ‘Download Button’ to prompt action), thus leveraging the primacy effect to encourage conversions.
At the bottom of the landing page, the Digital Marketing Institute includes graphics showcasing its top-level clients to create a sense of brand authority that sticks in the consumers’ mind while providing clear, concise FAQs in a clean dropdown format.
This recency effect-style approach ensures that visitors can recall essential details about the courses the DMI provides while remembering the impressive clients that brand has served.
Applying the serial position effect to your landing pages will give your UX design and content concepts definitive direction, improving navigation and boosting engagement in the process.
To build on the examples we’ve explored, here are some additional tips based on the serial position effect to help you improve your landing page UX:
Place your most expensive items or services at the top of your landing page to make your mid-range items or services appear less expensive and increase your average order value (AOV).
Add an alluring image, strapline, and CTA button to your top of page banner to deliver important information in a way that minimizes cognitive strain and increases consumer conversions.
Break up the text in the middle of the page with subheadings, images, bolded or italicized font, bullet points and small chunks of text to make your UX design more navigable. Doing so will also increase your chances of leading consumers to important information further down the page.
Position valuable information and USPs towards the bottom of the page and use informational CTA buttons to tell the user what to do next.
Always ensure that your landing page design is clean, logical, and easy to navigate. If you don’t put functionality first, it’s likely that your UX offerings will be poor and your visitors will not retain any information.
How to use experimentation in design
Applying effective design and copywriting principles to your various digital touchpoints while leveraging the serial position effect to deliver valuable information to your consumers will accelerate your commercial success.
But, in an increasingly saturated digital age where the consumer has a wealth of their fingertips, how do you know if your design and serial position effect-based efforts are working as they should?
A range of factors including color, layout, design elements, and even a consumer’s cognitive bias can impact landing page browsing behavior. So, the best way to understand if your initiatives are working and experiment with design effectively is though A/B testing. With a combination of effective data and the right A/B testing platform, it’s possible to pinpoint a specific landing page or user interface’s strengths or weaknesses.
By developing two versions of the same landing page, you can drill down into specific page elements and discover which performs best.
For example, you might find that version ‘A’ of a landing page is earning more engagement above the fold due to the design or placement of a ‘Shop Now’ button. Through testing, you might also find that version ‘B’ is converting more email subscribers as a result of a particular piece of copy or messaging.
If you hone in on this wealth of comparative information, you will gain the power to experiment with every design element imaginable, taking the best-performing elements to create a fully-optimized version of a specific page or touchpoint.
A/B testing will give your design experimentation activities shape while protecting your marketing budget.
If you understand which messaging or design elements to focus on, you can get to the root of the issue and make tweaks for optimizations that are likely to offer the best possible return on investment (ROI).
Concerning the serial position effect, through A/B testing and experimentation you will be able to flatten the serial position curve to balance the information on your interfaces or landing pages.
By balancing the information elements on your interfaces or landing pages, you can make your UX designs easier to navigate while improving brand engagement. You will also gain the ability to experiment with design elements to emphasize the information or assets featured at the top or bottom of your digital touchpoints.
Essentially, if users aren’t engaging with the information at the top or bottom of a specific page, it will become clear that your serial position effect-centric efforts aren’t working. From there, you can experiment with the hierarchy of your information in addition to design elements including buttons, color combinations, imagery, copy formatting, and text boxes.
At this point, it’s worth noting that in our ever-evolving commercial landscape, experimentation never stops. What works today may not tomorrow — and to optimize your digital touchpoints for sustainable growth, constant testing and evolution is essential.
“Design creates culture. Culture shapes values. Values determine the future.” — Robert L. Peters, Graphic Designer
Final thoughts
We’ve outlined the fundamentals of the serial position effect and looked at how to apply the concept to UX and landing page design while outlining the importance of experimentation and testing.
Reflecting on our journey, what is crystal clear is that, in order to deliver the very best designs and UX offerings to your consumers, you need to reduce cognitive strain as much as possible.
The serial position effect helps us to understand human limitations in terms of both long term and short term memory, as well as the importance of ordering your information effectively.
As designers, when applying the serial position effect, it’s critical to empower the user by providing task-relevant information on the screen where possible, sharing concise prompts or cues, reducing the level of recall needed across the user journey, and highlighting the most valuable information at the start and end of a sequence where necessary.
When interacting with your digital touchpoints or interfaces, your users shouldn’t be overwhelmed with information. They should be able to navigate every aspect of your interfaces or landing pages intuitively, with little additional thought, while understanding what to do next and why they are doing it.
Your UX and design offerings should deliver relevant, valuable information to your users in a way that is completely seamless — and, by using the serial position effect to guide your decision, you will set yourself apart from the competition.
At AB Tasty, security is fundamental to delivering the best-in-class customer experiences. This is not only a belief that we share with our clients but have also woven into the fabric of our company operations.
AB Tasty’s firm footing in the world of tech innovation entails upholding the highest standards of security and data protection. To that end, the company has successfully completed ISO 27001 certification — a feat that not only cements our level of excellence in information security but as well points to a strong privacy ecosystem.
The ISO 27001 certification, alongside our GDPR compliance as well as PCI-DSS and SOC 2 practices, is among the many layers of AB Tasty’s ambitious security program — designed for continuous review of how we handle sensitive data in our procedures and systems.
A people-first approach to security
Prior to establishing any policy or systematic implementation, the team knew they had to evangelize the central role of security. With that in mind, Chief Information Security Officer (CISO) Matthieu Chaignot united the IT, infrastructure and legal teams to be fervent defenders of security before all else.
“The truth is we are only as strong as our weakest link so we need to make sure that everyone understands the importance of security and be conscious of how they are coming across data or assets. It wasn’t enough to design the best processes or implement new tools, we needed to turn everyone into security addicts.”
Matthieu Chaignot, Chief Information Security Officer
The cross-functional approach to security ensured that while the tech and engineering experts were behind the deployment of all critical domains, the employees were the frontline protectors.
The road to ISO 27001 certification
The highly sought-after ISO 27001:2013, created by the International Organization for Standardization (ISO), represents the global standard in information security. Specifically, the group establishes guidelines on how companies manage their information systems and secure their assets.
AB Tasty’s client- and server-side experiments enable businesses to launch better products faster and drive more conversions, engagement and revenue across multiple platforms. The ISO 27001 certification of both products was fundamental to building on the trust we have with our customers.
The certification confirms that we have not only identified all the potential risks, but we have implemented the right information security practices to address those risks. The ISO 27001 certification ensures that we have:
Implemented IT security policies and procedures to uphold business continuity;
Mitigated risks through formalized security controls and countermeasures; and
Maintained and continually improved ISMS (Information Security Management System).
Building a reliable security infrastructure
When it comes to security, it’s more than a set of measures. It’s a mindset. The data protection of employees and clients is crucial to any successful business relationship. From our information security management systems to our products, they are built with the highest standards of protection. It also means that as our company continues to scale in volume, the security controls we have in place will become more robust over time.
“From the very beginning, AB Tasty has effortlessly worked to ensure privacy and compliance. The accreditations and industry-wide recognitions do not change our approach, but rather highlight our commitment to the security of external and internal data on a daily basis”.
When it comes to feature testing, you’re in a bind.
On the one hand, you need real-world data and feedback from real-world users. You know that every new feature you develop is, at best, an educated guess about what your real-world users want from you. No matter how educated that guess might be, and no matter much internal validation you perform, you can only generate meaningful data and feedback on each new feature you develop by releasing it to real-world users to test out in their real-world environments.
On the other hand, it’s risky to give real-world users an unproven feature. You know that every new feature you release might have something wrong with it. Maybe there’s a technical issue you missed during development. Maybe it just doesn’t align to user expectations as closely as you hoped. No matter the issue, releasing an unproven feature can cause real harm to your brand’s user relationships.
This is a tricky problem and one that is never going to be fully solved. But, thankfully, there are methods you can follow to minimize the problem, and collect real-world data and feedback while mitigating the impact when something (inevitably) goes wrong.
In this piece, we’ll explore one of these methods— rollbacks.
What is a Feature Rollback?
It’s a simple practice, with powerful implications.
When you perform a rollback, you take some code out of a live environment. Back in the day rollbacks could be truly massive. Software products used to be updated in giant new releases that could include a wide range of changes— including multiple new features and significant changes to existing features. If one of these huge releases had some fatal bugs in it, or just wasn’t well-received by users, then the entire thing might need to be rolled back (even if the issues were contained within just a few elements of the release).
All of this has changed with the adoption of Agile methodology. Releases keep getting smaller and more incremental, and so do rollbacks. Most modern Product Managers have adopted phased release plans, where they only release a single new or upgraded feature at a time— and often only to individual segments. And when modern Product Managers do release multiple new or upgraded features at once, the different features are separate from each other.
This evolution has changed the way rollbacks happen. After a new release, Product Managers can now isolate the individual feature(s) that have proven unfit for live usage and perform a targeted rollback on them, and them alone. The whole rollback process is now much faster, much nimbler, much more precise— and delivers much greater benefits.
Why Should Product Managers Perform Rollbacks?
When a Product Manager properly structures and deploys rollbacks, they improve their ability to test new features in a real-world environment with real-world users with a minimal level of risk. An imperfect feature is no longer the end of the world. If a feature has development issues or poor alignment with user requirements, you can perform a rollback and remove it from a live environment in real-time with just one click.
For Product Managers, this changes the game. The more mature your rollback capability, the more you can afford to make mistakes. Your risk shrinks, giving you the freedom to test more features with more users earlier in the development cycle, ultimately leading you to iterate your products faster and faster.
Now, rollbacks are not a silver bullet. They don’t absolve you from doing everything you can to develop the highest-quality features possible before you test them. But rollbacks allow you to test new features with greater confidence and reduced concerns about creating problems for your users.
When Should You Perform a Rollback? Two Common Use Cases
For most Product Managers, there are two common use cases why you might need to perform a rollback.
Rollback Use Case 1: Your Feature has a Bug
This first use case is pretty self-explanatory.
You might have the most robust and thorough QA and testing processes in the world. It’s still highly likely your new features will still have one or more bugs in them when you release them into a live environment. Maybe they’re issues you just didn’t think to search for or didn’t know how to search. Maybe they’re issues that only show up in live environment after hundreds of real-world users tool around with the feature.
Regardless of the reason, if significant technical issues pop up in your new feature, then you’ll likely want to perform a rollback on that feature to fix it. With the right rollback process, you can react to these errors in near-real-time and remove the feature—and maybe even fix it—in minutes before it impacts too many users.
Rollback Use Case 2: Your Feature is Poorly Received
This second use case is a little more sophisticated.
Essentially, after you release a new feature you monitor how users respond to it, and how well it’s hitting your business KPIs. If your new feature is not performing as expected, and is generating negative usage data and user feedback, then you can perform a rollback to remove it from its live environment. If it isn’t hitting—or at least tracking towards—its business goals, then it might not be worth keeping live.
After you roll back your feature, you can either utilize the data and feedback you collected to fix the feature and help it better align to user expectations and business requirements, or you can decide that the feature was fundamentally misguided and just needs to be retired.
With the right rollback process, you can also review and respond to the usage data and user feedback you receive in near-real-time, and prevent too many users from getting too disgruntled about receiving a feature that misses the mark.
What Do These Two Use Cases Have in Common?
In one word: speed.
In both use cases, rollbacks are most effective—and mitigate the most risk—when you are able to first monitor feature performance in real-time, to then translate that performance into a quick “yes/no” decision to rollback (or not), and finally to execute on that rollback decision as rapidly as possible.
The faster you can go through this entire process, the lower the chance that you will create a prolonged negative user experience. In some scenarios, the decision to perform a rollback and the execution of that rollback need to happen in minutes.
It’s a daunting mandate, but here are a few tips to help you meet it.
How to Make Faster Rollback Decisions
It’s challenging to decide—in the moment—whether or not to rollback a feature. Even the best feature release can be complex and chaotic.
There are multiple moving parts to monitor…
There are many different data and feedback points to take into consideration…
And there’s a lot of emotion at play…
You and your team just spent weeks, maybe even months, pouring your blood, sweat, and tears into designing and developing the new feature that you’re testing. If your users love it, then you get that sugar high of knowing you just completed a job well done, and you can just sit back and watch the good data and feedback roll in. But if your users don’t immediately respond as positively as you hoped, then it’s easy to experience an emotional crash and to want to rollback the feature before you even know if the bad response is consistent, let alone what you should do to fix your errors in the next iteration.
For these reasons, and many more, it’s hard to make the right rollback decision in the moment during a feature release test. Instead, it’s better to make your rollback decisions before you release your new features into the wild.
Here’s what we mean.
Basically, before you release any new features to any real-world users, you first decide what success and failure looks like for this feature in objective, data-driven terms.
Then, you decide how much data your release will need to generate before you can make an accurate call about whether the feature is a success or a failure.
Finally, you use these parameters during your release to make objective “yes/no” decisions about whether or not you should rollback your feature at any point. Instead of getting caught up in the moment, you just monitor the performance metrics that you decided were most important, and once they hit the thresholds you set prior to release you simply follow the plan and you either rollback the feature or you don’t— no real-time agonizing required.
How to Execute Rollbacks Faster
In the past, it was near-impossible to perform a rollback quickly from a technical perspective. You needed to have a technical team standing by, waiting to dig into the code to turn off live features, or to revert to a prior state of the entire platform. The entire process was slow, it was labor-intensive, and it took your technical teams away from their valuable development work.
Software has solved all of these problems. With our own feature management platform, you can rollback a feature in real-time by just toggling a single field with just one click. You don’t need any technical expertise to do so. You don’t need to develop and test a complex rollback process prior to feature release. You don’t even need to think about the technical details— you can save all of that thinking to create the right strategic decision trees that we outlined in the prior section.
AB Tasty also gives you—or any non-technical user—the ability to perform sophisticated feature releases and rollbacks. You can release multiple features at once, monitor how each feature is performing individually, and only rollback the features that aren’t delivering. You can roll out a feature to multiple user segments and only rollback that feature to the individual segments that aren’t responding well to it. We designed our server-side solution to make the execution of rollbacks faster, easier, and far more intuitive than they ever were before.
20 years ago, tech companies were hit with the ‘Agile Revolution’. The idea? Shipping working software every week or two would help teams deliver better products, even if this method implied more risk. In other words, the ‘move fast and break things’ mentality reigned.
But that was two decades ago. Today, agile is mainstream, and new philosophies, building on the agile movement, have come to the fore; namely, Continuous Integration, Delivery and Deployment, largely geared towards DevOps teams. Their big draw is that these processes and tools automate quality assessment, assuring that when code is merged in piecemeal fashion – and not on one big bang release day – it works. Even better, software can be deployed to the product environment at any time, by anyone. Now, your product manager can take the reins.
Today, the market is ready to go a step further. From Agile to Continuous Integration, Delivery and Deployment comes a thirst for Continuous Development. Continuous Development – we could even call it Continuous Activation – encompasses all of these ideas, but takes the logical next step. It puts even more control and autonomy in the hands of Product Managers. It allows them to not only deploy software themselves, (with mitigated risk), but also to pick and choose according to their own prerogatives which audiences are exposed to a given feature. In other words, they can run experiments, personalize the user experience, and exercise complete rollback control based on real-time data.
Continuous Development platforms and processes transform the Product Manager into a Chief Experimentation Officer, and there are many reasons to embrace this new paradigm shift:
Move Fast, Risk Less
‘Move fast and break things’ only works if you’re willing to accept the consequences of what you’ve broken. Most software developers would still like to move fast, but without the risk.
Continuous Development and the tools that support it factor in risk assessment. By avoiding code merges on one big release day, and by enabling progressive rollout techniques (canary deployment, ring deployment), developers can avoid putting all of their metaphorical eggs in one basket. If your system has a feature flagging or rollback KPI embedded in the platform, switching off a defective or negative feature can be done instantaneously and painlessly.
Your Customers, Not Your HIPPOs, Decide
How do decisions get made in your tech company? Chances are, HIPPOs, new bosses, vocal salespeople, consulting groups or the noisiest Product Manager in the room dominate that discussion, letting their personal experience, gut feeling or intuition determine the road map.
With Continuous Development platforms, the focus shifts from subjective ideas to customer feedback and data. Early adopter programs, beta testing, progressive deployment, A/B tests… all of these methods, enabled by feature flagging and other Continuous Development techniques, make your main measurement of success the behavior and opinions of your customers.
In a B2B context, this might look like extensive interviews with early adopters. In B2C, it’s likely your support teams or community manager who will pick up on positive or negative feedback around a new feature launch. Either way, Product Managers get direct access to the Voice of the Customer and can form data-driven arguments for why to rollback, stick with or modify a new feature.
Get off the Ford Line
If your team is project-driven, chances are your Product Managers and developers feel they need to keep their heads down and noses to the grindstone, working on their piece of the software production puzzle. They might be productive, they might be agile, but they might also not really feel the business impact of what they’re working on 40+ hours of the week.
When you can experiment with and test the features you’re developing; when you can get direct user feedback and adjust your work accordingly; when you have clear, measurable KPIs that determine success, your work all of a sudden feels a lot more meaningful. This keeps teams motivated, fresh and loyal.
Marketing and Product Manager Alignment
When you give your Product Managers more control, it’s easier for them to align with the teams around them, especially the Marketing and Communication departments. A new feature release, especially depending on the size and importance of your company, can mean a big web of marketing and communications campaigns. Emailings, press releases, articles, social network posting, corporate website updates…retroplannings and shifting deadlines are much easier to manage when your Product Managers are in the driver’s seat and not beholden to developer teams that have other priorities and are even more far removed from your marketing and communications personnel.
Developers Focus on Core Business Objectives
If you have a robust developer team, there’s a chance you could set up these types of feature management systems in-house, without the need for a dedicated platform. But this is time-consuming, and one could argue that it diverts skills and resources away from your core business objectives.
I believe that the time is now for Continuous Development. By turning our Product Managers into Experimenters, we’re able to build a better product and bring it to market faster, with less risk; we continue in the vein of ‘customer obsession’; we keep our teams creative and motivated; and we generally build up what, at AB Tasty, we’ve been advocating for since our founding – a test and learn, experimentation culture.
