Anaïs Levy shares how luxury brands use experimentation to balance brand image with business performance, plus why removing prices can actually boost conversions.
Anaïs Levy knows the secret behind what drives luxury digital experiences. As the E-commerce & Omnichannel Services Insights Manager at the Kering Group, the powerhouse group behind iconic luxury brands such as Gucci, Balenciaga, and Saint Laurent to name a few, she leverages over ten years in conversion rate optimization to help some of the largest luxury brands build dazzling digital experiences.
Before joining Kering, Anaïs worked across various industries, from travel giant Expedia to luxury group LVMH. Her unique position involves analyzing business performance across multiple luxury brands, helping them make data-driven decisions while respecting their distinct artistic visions. As a long-term AB Tasty customer and frequent speaker at industry events, Anaïs brings a rare perspective on how luxury brands navigate the delicate balance between brand image and business optimization.
Anaïs Levy spoke with AB Tasty’s Head of Marketing and host of “The 1000 Experiments Club” podcast, John Hughes, about navigating creative constraints in luxury experimentation, leveraging cross-brand benchmarking to drive results, and how omnichannel thinking is revolutionizing conversion rate optimization in the luxury space.
Here are some of the takeaways from their conversation.
Beyond the website: Luxury’s omnichannel reality
The numbers tell a story that completely reframes how we should think about luxury e-commerce: “Among all our visitors, if we sum up visitors in the stores and visitors on our websites, the majority, like 90% is coming to the website and 10% is our traffic in store,” Anaïs shares.
But here’s the twist—most purchases still happen offline. This flips traditional CRO thinking on its head.
“When you have this overview of understanding the business, understanding how luxury websites fit into the whole customer journey, conversion rate optimization is about understanding how you make the most of each asset you have,” she explains.
For luxury brands, websites serve multiple purposes beyond direct sales. Customers use them for research and discovery, to prepare for store visits, for aspirational browsing, and to access omnichannel services like click-and-collect or appointment booking.
“We know we have aspirational customers, but we were not treating them as a specific segment. So everybody coming to the website should in the end convert. And I guess now with the rise of omnichannel and services… we have come to this conclusion that a significant amount of traffic is not going to purchase online,” Anaïs notes.
The takeaway? Stop measuring luxury e-commerce success purely on online conversion rates. Instead, it’s about thinking bigger. How does your digital experience drive overall brand engagement and omnichannel revenue?
Discover the luxury industry’s glittering guide to all things optimization.
Working with luxury brands means constant negotiation between artistic vision and business performance. “It’s a lot of compromise,” Anaïs admits. “The brand image, the design is really the voice of the artistic director.”
But here’s where persistence pays off. When faced with a creative “no,” Anaïs doesn’t give up—she waits, gathers more data, and asks again. “You have to be stubborn because it could be a no. But two months, three months, six months after you ask again and one day you would have a yes,” she explains.
Her secret weapon? Benchmark data across sister brands. When one Kering brand achieves better checkout completion rates than another, it becomes harder to argue against proven improvements.
“If your sister brands can achieve these figures and they have kind of the same backbone, the same services, same offer, it means that there is something that we are not doing right,” she points out.
This creates a unique advantage where luxury brands can iterate on proven concepts while maintaining their distinct identities. Anaïs’s team has built AB test catalogs shared across all brands and runs group-wide experimentation events to facilitate this knowledge sharing.
Think global, analyze local with segmentation
Anaïs’s team runs experiments globally but analyzes results with surgical precision. “The tip I would give to people who would listen to this podcast is really think global, because then if you make it work for most of your users, your gains are way higher,” she advises.
But the magic happens in the analysis. “When you analyze, don’t forget to try looking at some important segments. So countries could be some… We are using a lot EmotionsAI segmentation to analyze the results because it gives you ideas about why the ‘Competition’ segment didn’t like these experiments,” she explains.
This granular approach reveals opportunities for personalization. By breaking down results by country, device, and behavioral and emotional segments, teams uncover insights that would be invisible in aggregate data.
The strategy works because it balances efficiency with insight: global rollouts maximize impact and streamline development, while segmented analysis reveals why certain groups respond differently, creating opportunities for follow-up experiments that target specific segments with tailored experiences.
What else can you learn from our conversation with Anaïs Levy?
The surprising price experiment: How removing prices from product listing pages actually increased conversions by focusing attention on products rather than cost
AI’s luxury future: From productivity tools to conversational search that mimics in-store personal shopping experiences
The newsletter discovery: How a failed lazy-loading test accidentally revealed hidden engagement opportunities in page footers
Cross-brand collaboration: The internal tools and processes that help luxury brands share learnings while maintaining their unique identities
About Anaïs Levy
Anaïs Levy has over ten years of experience in conversion rate optimization, spanning industries from travel (Expedia) to luxury (LVMH, Kering). At Kering Group, she manages business performance and insights across multiple luxury brands, including Gucci, Saint Laurent, and Balenciaga. Her unique role involves balancing data-driven optimization with the creative constraints of luxury brand management, making her a sought-after speaker on experimentation in highly regulated creative industries.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by John Hughes, Head of Marketing at AB Tasty. Join John as he sits down with the experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
Building a culture of experimentation requires an appetite for iteration, a fearless approach to failure and a test-and-learn mindset. The 1000 Experiments Club podcast digs into all of that and more with some of the most influential voices in the industry.
From CEOs and Founders to CRO Managers and more, these experts share the lessons they’ve learned throughout their careers in experimentation at top tech companies and insights on where the optimization industry is heading.
Whether you’re an A/B testing novice or a seasoned pro, here are some of our favorite influencers in CRO and experimentation that you should follow:
Ronny Kohavi
Ronny Kohavi, a pioneer in the field of experimentation, brings over three decades of experience in machine learning, controlled experiments, AI, and personalization.
He was a Vice President and Technical Fellow at Airbnb. Prior to that, he was Technical Fellow and Corporate Vice President at Microsoft, where he led the analysis and experimentation team (ExP). Before that, he was Director of Personalization and Data Mining at Amazon.
Ronny teaches an online interactive course on Accelerating Innovation with A/B Testing, which was attended by over 800 students
Ronny’s work has helped lay the foundation for modern online experimentation, influencing how some of the world’s biggest companies approach testing and decision-making.
He advocates for a gradual rollout approach over the typical 50/50 split at launch:
“One thing that turns out to be really useful is to start with a small ramp-up. Even if you plan to go to 50% control and 50% treatment, start at 2%. If something egregious happens—like a metric dropping by 10% instead of the 0.5% you’re monitoring for—you can detect it in near real time.”
This slow ramp-up helps teams catch critical issues early and protect user experience.
Talia Wolf is a conversion optimization specialist and founder & CEO of Getuplift, where she helps businesses boost revenue, leads, engagement, and sales through emotional targeting, persuasive design, and behavioral data.
She began her career at a social media agency, where she was introduced to CRO, then served as Marketing Director at monday.com before launching her first agency, Conversioner, in 2013.
Talia teaches companies to optimize their online presence using emotionally-driven strategies. She emphasizes that copy and visuals should address customers’ needs rather than focusing solely on the product.
For Talia, emotional marketing is inherently customer-centric and research-based. From there, experiments can be built into A/B testing platforms using a clear North Star metric—whether checkouts, sign-ups, or add-to-carts—to validate hypotheses and drive growth.
Elissa Quinby is the Head of Product Marketing at e-commerce acceleration platform Pattern, with a career rooted in retail, marketing, and customer experience.
Before joining Pattern, she led retail marketing as Senior Director at Quantum Metric. She began her career as an Assistant Buyer at American Eagle Outfitters, then spent two years at Google as a Digital Marketing Strategist. Elissa went on to spend eight years at Amazon, holding roles across marketing, program management, and product.
Elissa emphasizes the importance of starting small to build trust with new customers. “The goal is to offer value in exchange for data,” she explains, pointing to first-party data as the “secret sauce” behind many successful companies.
She encourages brands to experiment with creative ways of gathering customer information—always with trust at the center—so they can personalize experiences and deepen customer understanding over time.
Lukas Vermeer, Director of Experimentation at Vista, is an expert in designing, implementing, and scaling experimentation programs. He previously spent over eight years at Booking.com, where he held roles as a product manager, data scientist, and ultimately Director of Experimentation.
With a background in machine learning and AI, Lukas specializes in building the infrastructure and processes needed to scale testing and drive business growth. He also consults with companies to help them launch and accelerate their experimentation efforts.
Given today’s fast-changing environment, Lukas believes that roadmaps should be treated as flexible guides rather than rigid plans: “I think roadmaps aren’t necessarily bad, but they should acknowledge the fact that there is uncertainty. The deliverable should be clarifications of that uncertainty, rather than saying, ‘In two months, we’ll deliver feature XYZ.’”
Instead of promising final outcomes, Lukas emphasizes embracing uncertainty to make better, data-informed decisions.
Jonny Longden is the Chief Growth Officer at Speero, with over 17 years of experience improving websites through data and experimentation. He previously held senior roles at Boohoo Group, Journey Further, Sky, and Visa, where he led teams across experimentation, analytics, and digital product.
Jonny believes that smaller companies and startups—especially in their early, exploratory stages—stand to benefit the most from experimentation. Without testing, he argues, most ideas are unlikely to succeed.
“Without experimentation, your ideas are probably not going to work,” Jonny says. “The things that seem obvious often don’t deliver results, and the ideas that seem unlikely or even a bit silly can sometimes have the biggest impact.”
For Jonny, experimentation isn’t just a tactic—it’s the only reliable way to uncover what truly works and drive meaningful, data-backed progress.
Ruben de Boer is a Lead CRO Manager at Online Dialogue and founder of Conversion Ideas, with over 14 years of experience in data and optimization.
At Online Dialogue, he leads the team of Conversion Managers—developing skills, maintaining quality, and setting strategy and goals. Through his company, Conversion Ideas, Ruben helps people launch their careers in CRO and experimentation by offering accessible, high-quality courses and resources.
Ruben believes experimentation shouldn’t be judged solely by outcomes. “Roughly 25% of A/B tests result in a winner, meaning 75% of what’s built doesn’t get released—and that can feel like failure if you’re only focused on output,” he explains.
Instead, he urges teams to shift their focus to customer-centric insights. When the goal becomes understanding the user—not just releasing features—the entire purpose of experimentation evolves.
David Mannheim is a digital experience strategist with over 15 years of expertise helping brands like ASOS, Sports Direct, and Boots elevate their conversion strategies.
He is the CEO and founder of Made With Intent, focused on advancing innovative approaches to personalization through AI. Previously, he founded User Conversion, which became one of the UK’s largest independent CRO consultancies.
David recently authored a book exploring what he calls the missing element in modern personalization: the person. “Remember the first three syllables of personalization,” he says. “That often gets lost in data.”
He advocates for shifting focus from short-term gains to long-term customer value—emphasizing metrics like satisfaction, loyalty, and lifetime value over volume-based wins.
“More quality than quantity,” David explains, “and more recognition of the intangibles—not just the tangibles—puts brands in a much better place.”
Marianne Stjernvall has over a decade of experience in CRO and experimentation, having executed more than 500 A/B tests and helped over 30 organizations grow their testing programs.
Marianne is the founder of Queen of CRO and co-founder of ConversionHub, Sweden’s most senior CRO agency. As an established CRO consultant, she helps organizations build experimentation-led cultures grounded in data and continuous learning.
Marianne also teaches regularly, sharing her expertise on the full spectrum of CRO, A/B testing, and experimentation execution.
She stresses the importance of a centralized testing approach:
“If each department runs experiments in isolation, you risk making decisions based on three different data sets, since teams will be analyzing different types of data. Having clear ownership and a unified framework ensures the organization works cohesively with tests.”
Ben Labay is the CEO of Speero, blending academic rigor in statistics with deep expertise in customer experience and UX.
Holding degrees in Evolutionary Behavior and Conservation Research Science, Ben began his career as a staff researcher at the University of Texas, specializing in data modeling and research.
This foundation informs his work at Speero, where he helps organizations leverage customer data to make better decisions.
Ben emphasizes that insights should lead to action and reveal meaningful patterns. “Every agency and in-house team collects data and tests based on insights, but you can’t stop there.”
Passionate about advancing experimentation, Ben focuses on developing new models, applying game theory, and embracing bold innovation to uncover bigger, disruptive insights.
André Morys, CEO and founder of konversionsKRAFT, has nearly three decades of experience in experimentation, digital growth, and e-commerce optimization.
Fueled by a deep fascination with user and customer experience, André guides clients through the experimentation process using a blend of data, behavioral economics, consumer psychology, and qualitative research.
He believes the most valuable insights lie beneath the surface. “Most people underestimate the value of experimentation because of the factors that are hard to measure,” André explains.
“You cannot measure the influence of experimentation on your company’s culture, yet that impact may be ten times more important than the immediate uplift you create.”
This philosophy is central to his “digital experimentation framework,” which features his signature “Iceberg Model” to capture both measurable and intangible effects of testing.
Jeremy Epperson is the founder of Thetamark and has dedicated 14 years to conversion rate optimization and startup growth. He has worked with some of the fastest-growing unicorn startups in the world, researching, building, and implementing CRO programs for more than 150 growth-stage companies.
By gathering insights from diverse businesses, Jeremy has developed a data-driven approach to identify testing roadblocks, allowing him to optimize CRO processes and avoid the steep learning curves often associated with new launches.
In his interview, Jeremy emphasizes focusing on customer experience to drive growth. He explains, “We will do better as a business when we give the customer a better experience, make their life easier, simplify conversion, and eliminate the roadblocks that frustrate them and cause abandonment.”
His ultimate goal with experimentation is to create a seamless process from start to finish.
Chad Sanderson is the CEO and founder of Gable, a B2B data infrastructure SaaS company, and a renowned expert in digital experimentation and large-scale analysis.
He is also a product manager, public speaker, and writer who has lectured on topics such as the statistics of digital experimentation, advanced analysis techniques, and small-scale testing for small businesses.
Chad previously served as Senior Program Manager for Microsoft’s AI platform and was the Personalization Manager for Subway’s experimentation team.
He advises distinguishing between front-end (client-side) and back-end metrics before running experiments. Client-side metrics, such as revenue per transaction, are easier to track but may narrow focus to revenue growth alone.
“One set of metrics businesses mess up is relying only on client-side metrics like revenue per purchase,” Chad explains. “While revenue is important, focusing solely on it can drive decisions that overlook the overall impact of a feature.”
With a background in Global Business Management and Marketing, Computer Science, and Industrial Engineering, Carlos founded Floqq—Latin America’s largest online education marketplace.
In 2014, he founded Product School, now the global leader in Product Management training.
Carlos believes experimentation has become more accessible and essential for product managers. “You no longer need a background in data science or engineering to be effective,” he says.
He views product managers as central figures at the intersection of business, design, engineering, customer success, data, and sales. Success in this role requires skills in experimentation, roadmapping, data analysis, and prototyping—making experimentation a core competency in today’s product landscape.
Bhavik Patel is the Data Director at Huel, an AB Tasty customer, and the founder of CRAP Talks, a meetup series connecting CRO professionals across Conversion Rate, Analytics, and Product.
Previously, he served as Product Analytics & Experimentation Director at Lean Convert, where he led testing and optimization strategies for top brands. With deep expertise in personalization, experimentation, and data-driven decision-making, Bhavik helps teams evolve from basic A/B testing to strategic, high-impact programs.
With a focus on experimentation, personalization, and data-driven strategy, Bhavik leads teams in creating better digital experiences and smarter testing programs.
His philosophy centers on disruptive testing—bold experiments aimed at breaking past local maximums to deliver statistically meaningful results. “Once you’ve nailed the fundamentals, it’s time to make bigger bets,” he says.
Bhavik also stresses the importance of identifying the right problem before jumping to solutions: “The best solution for the wrong problem isn’t going to have any impact.”
Rand Fishkin is the co-founder and CEO of SparkToro, creators of audience research software designed to make audience insights accessible to all.
He also founded Moz and co-founded Inbound.org with Dharmesh Shah, which was later acquired by HubSpot in 2014. Rand is a frequent global keynote speaker on marketing and entrepreneurship, dedicated to helping people improve their marketing efforts.
Rand highlights the untapped potential in niche markets: “Many founders don’t consider the power of serving a small, focused group of people—maybe only a few thousand—who truly need their product. If you make it for them, they’ll love it. There’s tremendous opportunity there.”
A strong advocate for risk-taking and experimentation, Rand encourages marketers to identify where their audiences are and engage them directly there.
Shiva Manjunath is the Senior Web Product Manager of CRO at Motive and host of the podcast From A to B. With experience at companies like Gartner, Norwegian Cruise Line, and Edible, he’s spent years digging into user behavior and driving real results through experimentation.
Shiva is known for challenging the myth of “best practices,” emphasizing that optimization requires context, not checklists. “If what you believe is this best practice checklist nonsense, all CRO is just a checklist of tasks to do on your site. And that’s so incorrect,” he says.
At Gartner, a simplified form (typically seen as a CRO win) led to a drop in conversions, reinforcing his belief that true experimentation is about understanding why users act, not just what they do.
Through his work and podcast, Shiva aims to demystify CRO and encourage practitioners to think deeper, test smarter, and never stop asking questions.
E-commerce has completely changed the way shoppers interact with their favorite brands.
From the continued rise of mobile commerce to virtual-reality try-on tools and AI customer service, some consumer trends have proven to be evergreen while others fall out of fashion in a season. As e-commerce marketers, it can be hard to know when to chase a trend or stick to being consistent.
To help you better understand the mind of today’s consumers, we’ve broken down 10 key insights for e-commerce from our 2025 global report. Based on feedback from 4,000 consumers across the U.S., U.K., France, Italy, and Australia, this snapshot reveals how people discover new products, engage with AI, make purchase decisions, and much more.
1. Google Search is the first place for discovery
When it comes to starting an online shopping journey, Google Search is still king. Nearly two-thirds (63%) of global shoppers begin their hunt for a new product or service with a Google search.
This underscores the ongoing importance of SEO for e-commerce brands. If your product pages aren’t optimized, you risk missing out on a massive audience at the very first step of their journey.
2. Mobile takes over, but desktop still matters
By the end of 2024, smartphones accounted for nearly 80% of global retail site traffic and over two-thirds of online orders. Mobile is now the primary device for browsing and purchasing in categories like clothing, cosmetics, and entertainment.
However, desktop still plays a significant role in sectors such as travel and utilities, especially among older generations. Brands should continue to prioritize mobile-first design, but not neglect the desktop experience—especially for high-consideration purchases.
3. Millennials vs. Gen Z: Mobile app habits
Generational differences are shaping the future of e-commerce. For Gen Z, mobile apps are the second most popular starting point for shopping (48%), just behind Google. Millennials, on the other hand, split their preference between apps and brand websites (both at 35%). This means younger shoppers are more likely to use apps for discovery, while Millennials are equally comfortable with apps and direct website visits.
Brands need more than just a mobile presence to capture Gen Z’s attention. They need apps built for exploration, speed, and flexibility. With Feature Experimentation and Rollouts from AB Tasty, teams can continuously test and optimize in-app experiences without a full redeploy, ensuring their app evolves alongside user expectations.
4. Comparison shoppers lead the pack
Not all online shoppers are the same. Our research found that the most common shopper persona is “comparison-oriented”—30% of respondents compare multiple products before making a purchase. Only 11% identify as “speedy” shoppers who want to check out as quickly as possible. The rest fall somewhere in between, with 21% being “review-oriented,” 20% “confident,” and 18% “detail-oriented.” This diversity highlights the need for flexible site experiences that cater to different decision-making styles.
If one size doesn’t fit all, then understanding your audience is the first step to building experiences that truly convert.
5. Reviews are more influential than discounts or brand names
When it comes to influencing purchase decisions, high-quality reviews top the list globally. Shoppers trust peer validation more than discounts, convenience, or even brand names. Written testimonials and customer photos are especially valued, providing the authenticity and detail shoppers crave.
Make sure your reviews are visible, filterable, and packed with real customer insights to boost trust and conversions.
E-commerce moves fast. Get the insights that help you move faster. Download the 2025 report now.
Think you’re converting more by hitting new visitors with an email sign-up pop-up right away? Think again.
Too many pop-ups are the number one frustration for online shoppers worldwide, followed closely by slow-loading websites and difficulty finding products. While pop-ups can be effective for capturing leads or promoting offers, overuse can drive customers away. Use them strategically and ensure your site is fast and easy to navigate to keep shoppers engaged.
7. Loyalty is the key to better personalization
Personalization is more than just a buzzword—it’s a key driver of customer satisfaction and loyalty. The top way to make online shopping feel more personal, according to 35% of respondents, is by rewarding brand loyalty. Remembering preferences and suggesting relevant products also rank highly.
Brands that recognize and reward repeat customers with exclusive perks or early access to new products can turn shoppers into advocates.
8. AI adoption is growing, especially among younger shoppers
AI-powered tools like chatbots and virtual assistants are gaining traction, but there’s still room for improvement. Just under a quarter (23%) of shoppers have used AI tools and found them helpful, while 32% haven’t tried them but are open to it. Younger generations are more receptive: 32% of Gen Z and 30% of Millennials found AI tools helpful, compared to just 13% of Baby Boomers.
To win over skeptics, brands need to ensure AI support is fast, relevant, and seamlessly integrated with human assistance.
9. Shoppers just want frictionless experiences
When asked what would most improve their online shopping experience, the top answer was simple: removing frustrations like pop-ups, bugs, and broken pages. Tracking shipping, improving product search, and speeding up the shopping process were also highly valued.
Before investing in flashy features, brands should focus on getting the basics right—smooth, intuitive journeys are what keep customers coming back.
10. The gap between personalization and perception
Personalization is supposed to make shoppers feel seen—but only 1 in 10 consumers say their favorite brands truly “get” them. In fact, the most common answer was “somewhat,” as 39% of respondents said the messages and offers they receive are hit or miss. Another 34% said brands mostly deliver relevant content, but not always. For the majority, the digital experience feels inconsistent.
When personalization doesn’t land, it can come off as surface-level or even off-putting. The takeaway? Personalization isn’t just about using data—it’s about using it meaningfully, so relevance feels intentional, not accidental.
Conclusion
The bar for digital shopping experiences keeps rising, and today’s consumers are quicker than ever to click away when expectations aren’t met.
From discovery to checkout, each step in the customer journey has the potential to shape customer loyalty and long-term value. Our 2025 E-commerce Consumer report dives even deeper into generational trends, regional differences, and actionable strategies for optimizing your digital experience.
What is the future of experimentation? Bhavik Patel highlights the importance of strategic planning and innovation to achieve meaningful results.
A thought leader in the worlds of CRO and experimentation, Bhavik Patel founded popular UK-based meetup community, CRAP (Conversion Rate, Analytics, Product) Talks, seven years ago to fill a gap in the event market – opting to cover a broad range of optimization topics from CRO,data analysis, and product management to data science, marketing, and user experience.
After following his passion throughout the industry from acquisition growth marketing to experimentation and product analytics, Bhavik landed the role of Product Analytics & Experimentation Director at product measurement consultancy, Lean Convert, where his interests have converged. Here he is scaling a team and supporting their development in data and product thinking, as well as bringing analytical and experimentation excellence into the organization.
AB Tasty’s CMO Marylin Montoya spoke with Bhavik about the future of experimentation and how we might navigate the journey from the current mainstream approach to the potentialities of AI technology.
Here are some of the key takeaways from their conversation.
The evolution of experimentation: a scientific approach.
Delving straight to the heart of the conversation, Bhavik talks us through the evolution of A/B testing, from its roots in the scientific method, to recent and even current practices – which involve a lot of trial and error to test basic variables. When projecting into the future, we need to consider everything from people, to processes, and technology.
Until recently, conversion rate optimization has mostly been driven by marketing teams, with a focus on optimizing the basics such as headlines, buttons, and copy. Over the last few years, product development has started to become more data driven. Within the companies taking this approach, the product teams are the recipients of the A/B test results, but the people behind these tests are the analytical and data science teams, who are crafting new and advanced methods, from a statistical standpoint.
Rather than making a change on the homepage and trying to measure its impact on outcome metrics, such as sales or new customer acquisition, certain organizations are taking an alternative approach modeled by their data science teams: focusing on driving current user activity and then building new products based on that data.
The future of experimentation is born from an innovative mindset, but also requires critical thinking when it comes to planning experiments. Before a test goes live, we must consider the hypothesis that we’re testing, the outcome metric or leading indicators, how long we’re going to run it, and make sure that we have measurement capabilities in place. In short, the art of experimentation is transitioning from a marketing perspective to a science-based approach.
Why you need to level up your experiment design today.
While it may be a widespread challenge to shift the mindset around data and analyst teams from being cost centers to profit-enablement centers, the slowing economy might have a silver lining: people taking the experimentation process a lot more seriously.
We know that with proper research and design, an experiment can achieve a great ROI, and even prevent major losses when it comes to investing in new developments. However, it can be difficult to convince leadership of the impact, efficiency and potential growth derived from experimentation.
Given the current market, demonstrating the value of experimentation is more important than ever, as product and marketing teams can no longer afford to make mistakes by rolling out tests without validating them first, explains Bhavik.
Rather than watching your experiment fail slowly over time, it’s important to have a measurement framework in place: a baseline, a solid hypothesis, and a proper experiment design. With experimentation communities making up a small fraction of the overall industry, not everyone appreciates the ability to validate, quantify, and measure the impact of their work, however Bhavik hopes this will evolve in the near future.
Disruptive testing: high risk, high reward.
On the spectrum of innovation, at the very lowest end is incremental innovation, such as small tests and continuous improvements, which hits a local maximum very quickly. In order to break through that local maximum, you need to try something bolder: disruptive innovation.
When an organization is looking for bigger results, they need to switch out statistically significant micro-optimizations for experiments that will bring statistically meaningful results.
Once you’ve achieved better baseline practices – hypothesis writing, experiment design, and planning – it’s time to start making bigger bets and find other ways to measure it.
Now that you’re performing statistically meaningful tests, the final step in the evolution of experimentation is reverse-engineering solutions by identifying the right problem to solve. Bhavik explains that while we often focus on prioritizing solutions, by implementing various frameworks to estimate their reach and impact, we ought to take a step back and ask ourselves if we’re solving the right problem.
With a framework based on quality data and research, we can identify the right problem and then work on the solution, “because the best solution for the wrong problem isn’t going to have any impact,” says Bhavik.
What else can you learn from our conversation with Bhavik Patel?
The common drivers of experimentation and the importance of setting realistic expectations with expert guidance.
The role of A/B testing platforms in the future of experimentation: technology and interconnectivity.
The potential use of AI in experimentation: building, designing, analyzing, and reporting experiments, as well as predicting test outcomes.
The future of pricing: will AI enable dynamic pricing based on the customer’s behavior?
About Bhavik Patel
A seasoned CRO expert, Bhavik Patel is the Product Analytics & Experimentation Director at Lean Convert, leading a team of optimization specialists to create better online experiences for customers through experimentation, personalization, research, data, and analytics. In parallel, Bhavik is the founder of CRAP Talks, an acronym that stands for Conversion Rate, Analytics and Product, which unites CRO enthusiasts with thought leaders in the field through inspiring meetup events – where members share industry knowledge and ideas in an open-minded community.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by John Hughes, Head of Marketing at AB Tasty. Join John as he sits down with the experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
Talia Wolf reveals how emotional marketing can revolutionize your experimentation process and lift conversions.
Taking a customer-centric approach to marketing, founder and CEO of Getuplift, Talia Wolf, harnesses the power of emotional marketing techniques to increase visitor conversions.
Her natural interest in conversion rate optimization (CRO) and experimentation was sparked through her early work in a social media agency, later moving on to become an expert in the field – consulting for many companies on the subject, and speaking on stage at Google, MozCon and Search Love.
Guest host and AB Tasty’s Head of Growth Marketing UK, John Hughes, spoke with Talia about emotional marketing as a tool for optimization, delving into how customer research can facilitate the experimentation process, reduce the rate of failure, and earn the buy-in from company stakeholders.
Here are some of the key takeaways from their conversation.
What is emotional marketing?
Based upon the idea that emotion drives every single decision that we make in life, the emotional targeting methodology shifts the focus of your online marketing content from your solution, features, or pricing, to your customer. Rather than playing a guessing game and simply reshuffling elements on a page, this technique requires a deeper understanding of human behavior. By identifying customer intent and buying motivation, you can create an optimized experience, which meets their needs and increases conversions.
Backed by academic research, the fundamental role of emotion in our daily choices can be integrated into your strategy to better cater to your customers by figuring out a) their biggest challenges and, b) how they want to feel after finding a solution. What is their desired outcome?
With this in mind, you can optimize your digital communications with high-converting copy and visuals that speak directly to your customers’ needs. By shifting the conversation from the product to the customer, an incredible opportunity opens up to scale and multiply conversions.
How do you build and measure an emotion-based experiment?
Firstly, experimentation should be backed by research. From customer and visitor surveys, to review mining, social listening and emotional competitor analysis, Talia encourages extensive research in order to create the most likely hypothesis upon which to base an A/B test.
Once you know more about your customers, you can review the copy and visuals on your product page for example, and from your research you might discover that your content is not relevant to your target customer. You can then come up with a hypothesis based on their actual needs and interests supported by compelling social proof, and write a brief for your designer or copywriter based on the new information.
From there you can build your experiment into your A/B testing platform with a selected North star metric, whether it’s check-outs, sign-ups or add-to-carts, to prove or disprove your hypothesis. And, while we know that nine out of 10 A/B tests fail, emotional marketing facilitates the hypothesizing process, strengthening the chance of creating a winning experiment by testing variables that can actually impact the customer journey.
How to persuade stakeholders to support your experiments.
When it comes to CRO, there are often too many chefs in the kitchen, especially in smaller organizations where founders have a concrete vision of their customers and their messaging.
Talia explains that a research-based approach to experimentation can offer reassurance as part of a slow-and-steady strategy, backed by evidence. This personalized methodology involves talking to your customers and website visitors and scouring the web for conversations about your specific industry, rather than simply following your competitor’s lead.
It becomes a lot easier to propose a test to a founder or CEO when your hypothesis is supported by data and research, however, Talia recommends resisting the urge to change everything at once and rather, start small. Test the emotional marketing in your ads or send out an email sequence requiring only a copywriter, and share the results.
When you’re trying to get buy-in, you need to have a strong hypothesis paired with good research to prove that it makes sense. If this is the case, you can demonstrate the power of emotional marketing by running a couple of A/B tests: one where the control is the current solution-focused content and the variant is a customer-focused alternative, and another which highlights how customers feel right now versus how they want to feel – two important variations which help you to relate better to your customer. The key to garnering support is to take baby steps and continuously share your research and results.
What else can you learn from our conversation with Talia Wolf?
Why B2B purchases are more emotional than B2C. (15:50)
How to stand out in a crowded market by knowing your customer. (20:00)
How emotional marketing impacts the entire customer journey. (25:50)
How to relate to your customer and improve conversions. (32:40)
About Talia Wolf
Conversion optimization specialist Talia Wolf is the founder and CEO of Getuplift – a company that leverages optimization strategies such as emotional targeting, persuasive design, and behavioral data to help businesses generate more revenue, leads, engagement and sales.
Starting her career in a social media agency, where she was introduced to the concept of CRO, Talia went on to become the Marketing Director at monday.com, before launching her first conversion optimization agency, Conversioner, in 2013.
Today, with her proven strategy in hand, Talia teaches companies all over the world to optimize their online presence using emotional techniques.
About 1,000 Experiments Club
The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by John Hughes, Head of Marketing at AB Tasty. Join John as he sits down with the experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.
Black Friday – the time for sales and doorbusters and discounts – but what about the companies that are choosing not to participate this year? Rather than letting your site stay stagnant, you can still use this time to capitalize on the holiday traffic by testing new ideas for your site and gaining customer feedback in real-time.
There are many reasons why a company might opt to sit out of Black Friday sales. We’ve determined that most alternative Black Friday messaging falls into one of three major themes. These themes are:
Fair Friday/Do Good Friday: Brands take the time during the busy shopping season to instead promote donations or partner charities to help drive awareness of issues while building brand trust and developing corporate social responsibility.
Anti-Black Friday: Some brands choose to take not participating in Black Friday a step further by sharing messaging that challenges hyperconsumerism, promotes ethical consumption, or encourages a boycott of Black Friday. This might also take the form of a brand shutting down their e-commerce sales over the weekend in protest.
Normal Friday: Certain brands, including many luxury retailers, have not historically offered sales and discounts during Black Friday. In lieu of promoting active sales, these brands might instead drive traffic to new collections and releases for the season. This can also be an extension of Black Friday protests by choosing to conduct business as usual over the fanfare of the shopping season.
If you are considering an alternative Black Friday campaign this year, it is crucial to provide your visitors with clear communication about your campaign just like any other brand would do for their own sales and promotions.
Experimenting with easy-to-understand messaging, banners spread throughout the customer journey, and pop-ins reminding visitors of your initiative can help reduce confusion and provide a consistent experience for your customers.
Below, we have collected real-world examples of alternative Black Friday campaigns from AB Tasty clients and other brands who wanted to make the most out of the holiday traffic and gather important customer insights without compromising on their brand values.
Fair Friday / Do Good Friday Campaigns
L’Occitane
In addition to their Black Friday offers, the team at L’Occitane ran a “Give Back Friday” campaign in support of The Fred Hollows Foundation, a nonprofit organization focused on treating and preventing blindness and other vision problems. During the promotion period, $1 from every hand cream sold would be donated to the foundation.
Working within AB Tasty’s experimentation and personalization platform, custom code was created to support the campaign. The L’Occitane home page featured a large hero image along with a block of copy explaining the details of the promotion for customers.
Transparency in both the foundation that would receive the funds and for how the funds would be used to benefit others helped boost engagement and buy-in.
B&B Hotels
The team at B&B Hotels: Spain & Portugal set up a campaign on Black Friday called “Green Friday” where 5% of bookings made on that day would go to support the reconstruction of La Palma Island after intense fires.
They considered their users’ paths throughout the website when setting up their campaign. Clear messaging about the initiative was shared on the landing page and a reminder was displayed again at the booking stage to help push conversions. To reduce cart abandonment, an exit pop-in encouraged visitors to complete their booking before leaving the page.
Nature & Découvertes
As a part of their annual Fair Friday initiative, retailer Nature & Découvertes swapped out their standard hero image for one that highlighted their 2021 campaign for rewilding endangered habitats.
Throughout the last week of November, Nature & Découvertes encouraged both online and in-person customers to donate to the initiative by rounding up their purchase amounts to the next dollar. The brand then matched the total amount of donations.
Beyond the hero image swap on their homepage, Nature & Découvertes also drove traffic to a landing page explaining the initiative in greater detail. This helped customers have confidence that the initiative was genuine and offered the brand an opportunity to be transparent about the impact donations would have on wildlife and conservation efforts.
Anti-Black Friday Campaigns
REI Co-op
Outdoor recreation retailer REI Co-op’s #OptOutside initiative is a great example of brands taking a stance against Black Friday that also aligns with their brand values.
Instead of promoting a special sale or discount on Black Friday, REI Co-op chose to close their operations for their online store and drive visitors to learn more about the REI Cooperative Action Fund which works to make the outdoors more accessible for all communities.
With operations paused for the day, REI Co-op made sure to communicate clearly that no orders would ship out on that day by displaying a banner at the top of their pages. The simple banner helped reduce confusion for visitors and helped build positive attention for the brand by educating visitors about the #OptOutside mission.
Normal Friday Campaigns
BackMarket
BackMarket, the renewed devices retailer, does not offer additional sales or discounts on their products. In order to still capture attention from Black Friday visitors, they created a campaign dubbed “Any Old Friday”.
Taking inspiration from Black Friday doorbuster commercials, their landing page showed a new carousel image with copy that read “For an unlimited time only!”.
This cheeky take on the usual sales quips was a great opportunity for BackMarket to have a fun and engaging holiday season campaign without the need for offering discounts or sales.
Whether your brand wants to support an important cause or give visitors a break from Black Friday promotions, take advantage of increased traffic and drive engagement during the holiday shopping season with these creative alternatives to Black Friday.
On the fence about your Black Friday strategy? Get inspired by our e-book “How to Win Big During the E-Commerce Holiday Season”, featuring 30 experimentation and personalization ideas for Black Friday from AB Tasty clients around the world.
“Failure” can feel like a dirty word in the world of experimentation. Your team spends time thinking through a hypothesis, crafting a test, and finally when it rolls out … it falls flat. While it can feel daunting to see negative results from your a/b tests, you have gained valuable insights that can help you make data-driven, strategic decisions for your next experiment. Your “failure” becomes a learning opportunity.
Embracing the risk of negative results is a necessary part of building a culture of experimentation. On the first episode of the 1,000 Experiments Club podcast, Ronny Kohavi (formerly of Airbnb, Microsoft, and Amazon) shared that experimentation is a time where you will “fail fast and pivot fast.” As he learned while leading experimentation teams for the largest tech companies, your idea might fail. But it is your next idea that could be the solution you were seeking.
“There’s a lot to learn from these experiments: Did it work very well for the segment you were going after, but it affected another one? Learning what happened and why will lead to developing future strategies and being successful,” shares Ronny.
In order to build a culture of experimentation, you need to embrace the failures that come with it. By viewing negative results as learning opportunities, you build trust within your team and encourage them to seek creative solutions rather than playing it safe. Here are just a few benefits to embracing “failures” in experimentation:
Encourage curiosity: With AB Tasty, you can test your ideas quickly and easily. You can bypass lengthy implementations and complex coding. Every idea can be explored immediately and if it fails, you can get the next idea up and running without losing speed, saving you precious time and money.
Eliminate your risks without a blind rollout: Testing out changes on a few pages or with a small audience size can help you gather insights in a more controlled environment before planning larger-scale rollouts.
Strengthen hypotheses: It’s easy to fall prey to confirmation bias when you are afraid of failure. Testing out a hypothesis with a/b testing and receiving negative results confirms that your control is still your strongest performer, and you’ll have data to support the fact that you are moving in the right direction.
Validate existing positive results: Experimentation helps determine what small changes can drive a big impact with your audience. Comparing negative a/b test results against positive results for similar experiments can help to determine if the positive metrics stand the test of time, or if an isolated event caused skewed results.
In a controlled, time-limited environment, your experiment can help you learn very quickly if the changes you have made are going to support your hypothesis. Whether your experiment produces positive or negative results, you will gain valuable insights about your audience. As long as you are leveraging those new insights to build new hypotheses, your negative results will never be a “failure.” Instead, the biggest risk would be allowing a status quo continuing to go unchecked.
“Your ability to iterate quickly is a differentiation,” shares Ronny. “If you’re able to run more experiments and a certain percentage are pass/fail, this ability to try ideas is key.”
Below are some examples of real-world a/b tests and the crucial learnings that came from each experiment:
Lesson learned: Removing “Add to Basket” CTAs decreased conversion
In this experiment, our beauty/cosmetics client tested removing the “Add to Basket” CTA from their product pages. The idea behind this was to test if users would be more interested in clicking through to the individual pages, leading to a higher conversion rate. The results? While there was a 0.4% increase in visitors clicking “Add to Basket,” conversions were down by 2%. The team took this as proof that the original version of the website was working properly, and they were able to reinvest their time and effort into other projects.
Lesson learned: Busy form fields led to decreased leads
A banking client wanted to test if adjusting their standard request form would drive passage to step 2 and ultimately increase the number of leads from form submissions. The test focused on the mandatory business identification number field, adding a pop-up explaining what the field meant in the hopes of reducing form abandonment. The results? They saw a 22% decrease in leads as well as a 16% decrease in the number of visitors continuing to step 2 of the form. The team’s takeaways from this experiment were that in trying to be helpful and explain this field, their visitors were overwhelmed with information. The original version was the winner of this experiment, and the team saved themselves a huge potential loss from hardcoding the new form field.
The team at this beauty company designed an experiment to test whether displaying a message about product availability on the basket page would lead to an increase in conversions by appealing to the customer’s sense of FOMO. Instead, the results proved inconclusive. The conversion rate increased by 1%, but access to checkout and the average order value decreased by 2% and 0.7% respectively. The team determined that without the desired increase in their key metrics, it was not worth investing the time and resources needed to implement the change on the website. Instead, they leveraged their experiment data to help drive their website optimization roadmap and identify other areas of improvement.
Despite negative results, the teams in all three experiments leveraged these valuable insights to quickly readjust their strategy and identify other places for improvement on their website. By reframing the negative results of failed a/b tests into learning opportunities, the customer experience became their driver for innovation instead of untested ideas from an echo chamber.
Jeff Copetas, VP of E-Commerce & Digital at Avid, stresses the importance of figuring out who you are listening to when building out an experimentation roadmap. “[At Avid] we had to move from a mindset of ‘I think …’ to ‘let’s test and learn,’ by taking the albatross of opinions out of our decision-making process,” Jeff recalls. “You can make a pretty website, but if it doesn’t perform well and you’re not learning what drives conversion, then all you have is a pretty website that doesn’t perform.”
Through testing you are collecting data on how customers are experiencing your website, which will always prove to be more valuable than never testing the status quo. Are you seeking inspiration for your next experiment? We’ve gathered insights from 50 trusted brands around the world to understand the tests they’ve tried, the lessons they’ve learned, and the successes they’ve had.
