DirectAsia Simplifies Insurance Experiences with Empathy
+14.8%Access to Quote Page V1
+10.9%Access to Quote Page V2
Through the use of simplified and confidence-building journeys, DirectAsia is transforming the traditionally tedious task of buying vehicle and travel insurance into a seamless experience that reflects the overall convenience of their brand and services.
Part of this transformation lies in their strategic partnership with AB Tasty and the integration of EmotionsAI to better understand their customers and boost their experience optimization roadmap.
With insurance buyers increasingly seeking reassurance, trust, and intuitive experiences, DirectAsia recognized the need to evolve beyond traditional approaches to meet these demands. Like many financial services, insurance is inherently complex by nature.
DirectAsia ran tests focusing on an area on their website where they knew they had room for improvement. They compared the results of that test on segments selected by EmotionsAI versus a broader audience. Download the case study to find out how EmotionsAI expedited visitor journeys through to their quote page.
Spark your curiosity!
Get your copy of "DirectAsia Simplifies Insurance Experiences with Empathy" now.
CleanChoice Energy is a leading 100% renewable energy supplier in the U.S., building solar farms and providing consumers with alternative ways to access clean energy.
CHALLENGE
While the CleanChoice team’s paid media campaigns drove heavily engaged traffic to the website, they found those visitors weren’t quite ready to commit to pursuing clean energy.
In an effort to boost the performance of top-of-funnel and awareness campaigns, they decided to test if a timed pop-in on certain pages would drive better lead capture.
TEST IDEAS
The CleanChoice Energy team hypothesized that displaying an email pop-in within 15 seconds of users landing on one of three different test pages would generate more email sign-ups for their newsletter campaigns than its original placement as a button at the bottom of each page.
RESULTS
On average, the tests resulted in a 204% increase in email submissions with the pop-in than without. This helped increase the performance of the top-of-funnel awareness campaigns significantly.
TAKEAWAYS
Not every website visitor is ready to immediately make a purchase. This test proved to the CleanChoice Energy team that they could benefit from a lead-capture strategy for visitors across all stages of the funnel. The team made this pop-in a permanent feature of the site and began testing pop-ins with different messaging on other site pages to target visitors across additional funnel stages.
Spark your curiosity!
Get your copy of "CleanChoice Energy Drives Lead Gen with Pop-In" now.
Patyka Boosts Conversions and Enhances UX with AB Tasty
13%Visitors who see recommendations use them
2xUsers of recently viewed products spend 2x as much as other
Discover how Patyka, a pioneer in organic and natural cosmetics, transformed its e-commerce platform with AB Tasty’s personalized user experience solutions. As a leading brand in the rapidly growing organic beauty market, Patyka sought to elevate its online presence to match its esteemed reputation in physical stores.
In 2022, Patyka undertook a digital acceleration project, focusing on optimizing their user experience to increase conversions and sales. By implementing AB Tasty’s intelligent e-merchandising, personalized product recommendations, and automated features like “recently viewed products” and “abandoned cart recovery,” Patyka significantly improved the relevance and fluidity of its customer journey.
The results were impressive—13% of visitors exposed to product recommendations used them, and users of the “recently viewed” products feature spent twice as much as others. Additionally, the automation of manual tasks freed up valuable time for Patyka’s teams to focus on high-value projects, further enhancing brand image and customer satisfaction.
Download the full case study now to learn how AB Tasty can help your brand achieve similar success.
Spark your curiosity!
Get your copy of "Patyka Boosts Conversions and Enhances UX with AB Tasty" now.
Pan Pacific Hotels Group Uplifts Revenue with Audience Segmentation
+35.3%Visitor-to-book rate
+32.1%Overall search-to-select rate
Founded in 1975 and headquartered in Singapore, The Pan Pacific Hotels Group operates over 20 luxury hotels, resorts and serviced suites across Asia, North America, Oceania and Europe. The hospitality giant has stepped foot into the world of personalization, and AB Tasty is their platform of choice.
CHALLENGE
As we moved past the Covid pandemic, hospitality brands have been hungry to gain back lost revenue. It is key for brands to level up their customer experience (CX) to withstand intense competition.
TEST IDEA
For this experiment, the team was interested in uplifting overall bookings from couples and families. They theorized that creating a linked end-to-end experience for this audience group would uplift revenue, and AB Tasty was the perfect tool to support their personalization goals.
Two audience profiles were created from multiple data sources in a customer data platform and shared with the front-end marketing tools. These audiences were Families and Couples.
Using this audience data, a personalized digital experience was powered. Website visitors were directed to custom landing pages, which featured banners and offers tailored for their segment.
Family Variant
Couple Variant
RESULTS
The results were impressive. The team saw a 35.3% uplift in visitor-to-book and a 32.1% increase in search-to-select.
Couples and families who saw a personalized experience were 21% and 31% more likely to revisit the site respectively, compared to the control group.
Clicks on the homepage hero banner and navigation usage dropped significantly, proving that personalization enabled users to convert without extra website navigation.
Spark your curiosity!
Get your copy of "Pan Pacific Hotels Group Uplifts Revenue with Audience Segmentation" now.
Shiseido revitalizes experience optimization strategy with AB Tasty
6Markets Testing
The cosmetics giant has perfected the blend of beauty and data science.
Now Cynthia Bevilacqua, who leads EMEA e-commerce CRO & digital product roadmaps, wants to harness it.
6 Markets Testing
11 tests per month
1 Year WIth AB Tasty
When it comes to the beauty industry, competition is fierce. New brands pop up every day and that means legacy brands need to evolve or risk getting left on the shelf.
With a company philosophy rooted in the artful blend between beauty and science, Shiseido has fully embraced digital transformation in pursuit of being the leader in the data-driven beauty industry.
Today, the company works with AB Tasty to optimize e-commerce experiences for brand Shiseido and NARS in the EMEA region. Just one year into the partnership, Shiseido is now actively running around 11 tests per month across six markets.
But it wasn’t always like this.
At the start of 2022, Shiseido wanted to transform their existing but constricted experimentation strategy into an intuitive and scalable optimization program. The product team was eager to reach a new level of A/B testing capabilities as well as better optimize the time and resources spent on implementation and execution.
Cynthia Bevilacqua, digital product and user experience manager at Shiseido, knew it was time for a change. “This was March 2022, so the CRO mindset was new. We were doing some A/B tests, but it was limited to three or four tests in one year,” she recalls. “Up until this point, optimization was not a core focus. And I really wanted to change that.”
Place alternative payment options closer to “add to cart”
Offering payment installments is one way to motivate users to add more items to their cart. Even if a user doesn’t take advantage of the offer, having the option helps alleviate hesitancy around pricing.
The team at Shiseido wondered whether moving the Klarna payment copy block closer to the “add to cart” CTA could improve the click rate. With an A/B test targeted to the NARS and brand Shiseido UK markets, the team tested placing the price, Klarna copy and CTA closer together.
This small layout change led to a massive improvement with 129% increased clicks on “add to cart” for mobile and 159% for desktop.
ORIGINAL / VARIATION
Mobile
129% increased clicks on “add to cart”
24% increased clicks across all payment options
Desktop
159% increased clicks on “add to cart”
72% increased clicks across all payment options
New partner. New mindset.
A large global footprint requires a lot of cross-team collaboration. From their vantage point within the regional division, Cynthia’s team fields incoming requests from their local markets while also balancing strategic brand ideas from the global HQ in Tokyo.
With oversight on A/B testing and CRO strategies for the Shiseido EMEA region, Cynthia’s next hurdle would be setting a strategy for experimentation across each of the region’s unique markets. More frequent testing enabled by AB Tasty meant simply replicating any test in a new country wouldn’t cut it if they wanted to grow strategically.
“When we made the switch to AB Tasty, we wanted a partner who could help us with specialized testing in each market. We knew a one-size-fits-all did not work for our brand. You cannot say that because a test works well in the UK, that it will equally be successful in France or Germany because the consumer behavior can be quite different across each market.”
To sift through the prioritization of rolling out tests to new regions, Cynthia leveraged AB Tasty’s testing capabilities to implement internal processes. First, evaluating hypotheses through testing and data. Then, successful tests are added to the development roadmap, while the learnings from unsuccessful ones are shared with regional stakeholders and a new round of iterations starts.
“This process helps us ensure that we are not wasting valuable resources and bandwidth. AB Tasty not only enables us to quickly validate the hypothesis with data, but they also help us to say ‘no’ to a request because the data shows it’s not beneficial to the end customers.”
“AB Tasty not only enables us to quickly validate the hypothesis with data, but they also help us to say ‘no’ to a request because the data shows it’s not beneficial to the end customers.”
Cynthia Bevilacqua Digital product and user experience manager at Shiseido
Boost engagement with a higher visibility virtual try-on CTA
Shiseido collected insights that showed customers who interacted with a feature during their decision-making process had a tendency to convert.
Focusing on the French market, the team set up a variation giving the virtual try-on CTA a complete makeover. A small image, camera icon and brighter button color helped increase the CTA’s visibility on the product page.
Compared to the original CTA (which saw an average of 10% click rate), the revamped virtual try-on button saw a whopping 95% increase in clicks. “Add to cart” clicks also increased by 73% for those who used the feature.
ORIGINAL / VARIATION
95% clicks on virtual try-on
73% clicks on “add to cart” for virtual try-on users
16% overall clicks on “add to cart”
61% in transaction rate
Celebrating a culture of experimentation
After a year of working with AB Tasty, the team at Shiseido is thrilled by the tangible results and internal adoption of a CRO mindset. Since moving from four tests per year to over 10 tests per month with AB Tasty, Shiseido has significantly improved the customer experience. The success from 2022’s experimentation rollout resulted in an opportunity for Cynthia to grow the expertise of her team notably on strategic decision-making and roadmap management.
“The partnership we have with AB Tasty’s customer success team is great because we strategize together about our objectives. They feed us different ideas for testing and understand what’s working from other clients. It is fuel to get us thinking about our roadmap and what we should try to implement next.”
The experimentation mindset also fits perfectly into Shiseido’s desire to be a key player within data-driven beauty companies. With stakeholders invested in the testing mindset, Cynthia’s team is empowered to continue putting ideas to the test first before rolling them into a development roadmap.
“The test-and-learn approach is important to have because it’s not about what you think. ‘I think’ is not something we can use. We need data to tell that story,” Cynthia explains. “And now, this is our new reality with AB Tasty.”
WINNING TOGETHER 5 tips for operational excellence
It’s not easy (or practical) to tell someone their opinion is wrong. That’s why Cynthia Bevilacqua, digital product and user experience manager at Shiseido, advocates for using experience optimization platforms like AB Tasty to deliver data-driven decision-making from individual team members all the way up to stakeholders. It’s essential to sustainable growth.
Here are five tips to optimize team operations:
Monthly team meetings for individual tests and results
Quarterly stakeholder meetings for KPI deep dives and macro-insights
Clear KPIs and tracking dashboard
Gradual data-backed rollout to other relevant teams
Business performance or customer pain points at the center of every test
Deepening the partnership with AB Tasty
One learning that Cynthia’s team is incorporating into their 2023 strategy is giving tests more room to breathe in order to collect better data. Launching fewer tests and running them for a longer duration of six weeks with a minimum detectable effect (MDE) calculation allows the team to gather accurate data to inform the actions for the next iteration.
Entering year two of the partnership with AB Tasty, Cynthia is looking to unlock the next level of experience optimization by expanding the solution to Shiseido’s other EMEA brands. For Shiseido’s mature markets, the team plans to introduce personalization campaigns and build more mature testing. And by encouraging digital to local market teams to incorporate experimentation and the testing-first mindset, Cynthia is further driving Shiseido’s data-driven beauty company vision.
“Initially, we were focused on setting up the team for success through processes and an embedded CRO strategy. Now, we’re ready to accelerate with AB Tasty by using the data intelligence from the past year to drive our global experimentation program,” Cynthia says. “We won’t test just anything — we need to always make sure our tests are addressing a user pain point.”
Spreedly is a B2B payment orchestration platform that supports clients transacting through multiple payment services.
CHALLENGE
After Spreedly engaged AB Tasty as its experimentation platform of choice, we partnered with CXperts to fully onboard Spreedly in an effort to not only begin initial tests but also to build and support a mature experimentation program to scale as they grow.
TEST IDEA
CXperts and Spreedly decided to first test the homepage, hypothesizing that site visitors may not know about Spreedly’s trial. Additionally, improving the CTA could boost engagement and conversions. The teams developed two variations to test: one offering a “trial” and the other offering a “free trial.”
RESULTS
With this test, Spreedly realized a 34% increase in contact form submissions – their main goal – with 90% statistical significance. Although clicks to the CTA didn’t increase, it was clear that exposure to “trial” messaging had a positive impact.
AB Tasty has been critical in helping us make strategic, data-driven decisions that drive more conversions of our web traffic to leads for our sales team.
– Mike Taberksi , Senior Customer Insights Analyst
TAKEAWAYS
Over the course of the onboarding, CXperts was able to use AB Tasty to improve other key engagement metrics for Spreedly, such as pages visited, scroll depth, time on site, and revisit rate, all suggesting an increase in potential long-tail conversions. Additionally, the Spreedly team now has a roadmap of tests and the expertise they need to continue growing the program.
Spark your curiosity!
Get your copy of "Spreedly Boosts Lead Generation with AB Tasty" now.
King Living Increases Revenue 15% by A/B Testing on Site
+15%Clicks
+15% Overall Revenue
King Living is an award-winning Australian furniture retailer that has made a name for itself through selling bespoke items. After noticing a substantial number of drop-offs, King Living utilised AB Tasty to run two experiments to better understand consumers’ purchase behaviour, and, in turn, make data-driven decisions to improve user experience and generate more sales.
Improve product findability for customers
King Living’s website was very content heavy, and customers often had to navigate through content blocks before reaching the product. The team hypothesized that this resulted in drop-offs because it took too long for customers to find what they were looking for.
With this hypothesis in mind, King Living chose to experiment by hiding secondary navigation and displaying products before content. This move resulted in 15% more clicks on product images and 90% increased revenue for all of King Living’s ‘Jasper’ packages.
Consumers are more likely to take action if the information they are seeking is clearly presented to them. In this case, images were more effective than descriptions in communicating what the product was.
Through this experiment, King Living also discovered that their customers do not need a button to go to ‘Products’, which influenced the company’s Product Listing Page and Product Detailing Page redesign.
We noticed that our users don’t actually need a button to go to ‘Products’, so that’s influencing our design decisions in the whole PLP, PDP redesign.
Improve customer conversion by building stronger call-to-action
King Living noticed a substantial number of drop-offs on their check-out page, and pinpointed a few areas for improvement regarding their call-to-action (CTA) – inactive appearance, usage of internal speak and lack of focal point. These factors combined resulted in a CTA difficult for customers to comprehend.
In this experiment, they bolded the important points and translated internal speak into understandable language for consumers. King Living also made their call-to-action appear more visible by shifting away from grey, which is known as the colour of inaction. Through this experiment, King Living’s revenue grew by an astounding 15%.
Having an engaging CTA is important, even at the end of the purchase journey. Consumers are more compelled to follow through with their purchase when the CTA “speaks” to them. This makes a huge difference in minimising drop-offs.
“We had a 15% increase in revenue while that test was running – millions of dollars when you look at it in an annual sense,” Vanja says.
We had a 15% increase in revenue while that test was running — millions of dollars when you look at it in an annual sense.
Experimentation via AB Tasty helped King Living minimise drop-offs, increase conversions and grow business revenue.
Spark your curiosity!
Get your copy of "King Living Increases Revenue 15% by A/B Testing on Site" now.
How Hanna Andersson Experiments Its Way to Higher Lifetime Value
+8% PLP Clicks
+22% Purchase Rate
About Hanna Andersson
Adored by parents around the world, Hanna Andersson offers a wide range of children’s clothing and matching pajamas for the entire family. Founded in Portland, OR by Gun Denhart in 1983 as a purely catalogue company, the brand has since successfully made the transition to online sales.
Increasing Customer Lifetime Value
“One of our challenges is marketing our toddler collection,” explains Morganne Hatfield, Marketing Analyst at Hanna Andersson. “We recently combined the kids line and the toddler line into one collection. But toddler is a key demographic – if we can get moms to discover and fall in love with Hanna Andersson clothing for their young children, chances are they’ll stay loyal customers as their kids grow, increasing their overall customer lifetime value (CLV).”
Creating an e-commerce growth engine
Expanding a product line is a far-reaching goal that affects all aspects of a business. Morganne’s challenge was to figure out how to use Hanna Andersson’s website to increase awareness and sales. But where to start! The team could rewrite product descriptions, increase or decrease product image sizes, add customer reviews or implement a cross-sell bar…
The possibilities were endless: what was the best tactic to increase growth?
Experiment to increase uplift
Enter experimentation. Hanna Andersson’s SVP of Digital was a firm believer in the merits of testing and using data-driven rationales for optimization. Through a prioritized experimentation roadmap, the team could systematically test their optimization ideas, learn from the results, and implement winning variations. No more guesswork or time lost in internal debates.
Is a picture worth a thousand conversions?
One idea Morganne was eager to test involved the default image shown on the product listing pages (PLPs) and product description pages (PDPs) of the toddler section.
When someone went browsing through the toddler collection, the first image on the page showed the clothing item sized for an older child – not a 1 to 3 year old. Though the product itself was identical, it certainly didn’t look the same when sized for a lanky seven year old versus a toddler still round with baby fat. Her idea was to test whether it was worth the effort to photograph the smaller sizes and use them for the toddler product page images. Would they convert better?
Morganne decided to test swapping out the default image with one where the item was sized for a toddler, but only if the user was navigating from the toddler section of the site. “I wanted to test if buyers would react more positively to the age-appropriate clothing, whether this would increase basket rate and overall conversion rate.”
Control
Test
Significant revenue impact
After a quick set up, the test was live and results rolled in: the experiment was an absolute success. “I was honestly surprised by just how well it worked – clicks on PLPs increased by about 8%, and purchase rate on those items by 22%, which is huge. At the time of this test we only had a handful of toddler-sized items online. But we’ll soon be offering 100% of our line in these sizes, so if you multiply these results by hundreds of products, the revenue impact is significant.”
“Our merchandising team was thrilled”
Not only did the test have a positive impact on immediate business metrics, it was also another step in solidifying the marketing for the toddler clothing line.
“When I shared these results in our quarterly review, our merchandising team was thrilled,” explained Morganne. “Having this kind of data-driven evidence helps us confidently make strategic investments – this knowledge is really helpful,” concludes Morganne.
AB Tasty enables agility
AB Tasty’s ease of use was instrumental in the success of this experiment. Without the marketing-friendly interface and widget library, it wouldn’t have been possible to get this test up and running in under a week.
“It’s great to be able to use AB Tasty to launch these kinds of campaigns independently, without having to bother our dev team,” says Morganne. “We’re really pleased with the results we’ve had, and how agile the AB Tasty Customer Success team has been. The widget library especially – the countdown banners, NPS surveys – have been key in enabling us to be autonomous and launch campaigns fast. ”
Looking forward
This test on the toddler line was undoubtedly a success. But that doesn’t mean the testing ends. Continuous optimization is key to fully reaping the benefits of experimentation and personalization, and Morganne intends to continue perfecting the customer journey. “We dug into the data and realized that, although 70% of our traffic comes from mobile devices, there’s still a lot of room for improvement when it comes to converting that traffic,” explains Morganne. “We’re excited to see what more we can achieve with AB Tasty!”
Spark your curiosity!
Get your copy of "How Hanna Andersson Experiments Its Way to Higher Lifetime Value" now.
AB Tasty Enables MoneySmart to Innovate at the Pace of Their Ideas
+12%CTA clicks
+40%Use of filters
MoneySmart Group is South East Asia’s leading personal finance portal, helping consumers compare loans, insurance, and credit cards. From everyday decisions like choosing the best credit card to big decisions like buying a home, MoneySmart shows their customers what’s out there, recommends what’s best for them and helps with the application process.
AB Tasty accelerates experimentation
Today, MoneySmart’s product team can get an experiment up and running – and see results – in under 24 hours. Someone has an idea to test a form length, button color or banner? No sweat! But it wasn’t always like this, as Vincent Paca, Engineering Manager at MoneySmart, explains:
“Before AB Tasty, we were running our own, experimental platform. A lot of the developers’ time was spent creating and coding in different variations of tests. What this meant was that you could spend a sprint just to launch a very simple test. It was a very costly endeavor.”
Sanjiv Shah, Product Manager at MoneySmart, elaborates: “It took a long time to set up a test, it was very manual and painful. While we wanted to experiment with speed, it just took such a long time.”
This sluggish process is less than ideal.
First, testing inertia can mean missed opportunities to convert traffic to leads, or leads into customers. This can eat away at a business’s bottom line. Perhaps more crucially however, it can also entail falling behind competitors who are more rapidly refining the experience on their websites. Consumers increasingly expect financial sector brands to offer digital experiences that are as smooth and delightful as those of e-commerce or social media sites. If MoneySmart wanted to keep up, it was time to make a change.
That change came in the form of AB Tasty.
No such thing as a losing test
“What AB Tasty gave us was a tool that let us easily make changes to our website, deploy these changes and get the results out of the door within a day,” explains Vincent. “The QA function is fantastic,” adds Sanjiv. “You can actually see what a test will really look like for your customers by doing a test in production on just your computer’s IP address.” The best part? No flickering effect and virtually no change to the load speed time.
AB Tasty’s arrival was a watershed moment for the product and tech teams at MoneySmart.
Gone was the manual, tedious process of aligning devops and data teams to regulate traffic flows and build dashboards from scratch. Gone, too, was the stress of not knowing if a new idea would be a hit or a flop. “With AB Tasty, even if a test idea doesn’t ‘win’, we still save a lot on production cost and avoid frustration from developers, product managers and designers,” concludes Efrim Bartosik, Head of Product Design.
Teams come together around experimentation
One of the most appreciated aspects of AB Tasty, according to Bui Hanh, Product Manager at MoneySmart, is the perfect balance between features for technical and non-technical profiles. Not only is MoneySmart’s product team champing at the bit to run a multitude of tests, but so are the commercial, marketing and content teams. Through a testing tool that’s accessible to everyone, a true culture of experimentation is starting to take root, aligning departments around common goals and igniting growth.
A perfect example is the test shown below. Design, product and developer teams worked together to test a new layout for the ‘apply for a personal loan’ feature. The goal was to increase clicks to the ‘Apply Now’ CTA, and the new version did just that. CTR increased by nearly 12%, as did the use of the filters function (by 40%).
Takeaway
When barriers to experimentation – whether they be in terms of time, money or company culture – are removed, growth flourishes. Aligning teams around creativity and agility is a key ingredient to increasing both business outcomes as well as customer satisfaction. AB Tasty is the perfect partner to help product and marketing teams achieve both.
Spark your curiosity!
Get your copy of "AB Tasty Enables MoneySmart to Innovate at the Pace of Their Ideas" now.
OUI.sncf Relies on AB Tasty for Risk-Free Innovation
61%Increase in conversion rate
Fasterinnovation
For Hélène Doré, brand loyalty is her bread and butter.
As a Conversion Rate Optimization Manager at OUI.sncf, her day-to-day consists of thinking up ways to make customer experiences smoother and conversions higher. It’s a job she loves, but it’s not without its challenges. Today’s consumers are fickle and their expectations are exacting; one poor experience on an app or website can create a long-lasting negative feeling, potentially spiralling into poor brand perception, a drop off to a competitor, or bad word of mouth.
“More likely than not, when a customer faces a problem during his/her visit, it creates a bad memory, which may mean the customer doesn’t return. Creating an exceptional, personalized, and fluid customer experience gives a sentiment of recognition to the customer, which will incite him/her to complete a transaction and potentially come back. If we work on creating the most seamless and personalized customer journey, over time, this recognition will transform into loyalty and create stickiness, inciting the customer to come back and therefore form a virtuous relationship with us.”
– Hélène Doré, Conversion Rate Optimization Manager at OUI.sncf
The conversion rate optimization team, along with the product feature teams, at OUI.sncf has plenty of ideas for how to avoid these scenarios and create truly exceptional customer experiences. The question is: which ideas should they implement? Which should they scale? And how can they be sure an idea won’t backfire?
A/B Testing: The way of data, not debates
What Hélène and her manager Sunny Song, Lead CRO, needed was the ability to run data-driven experiments in order to know which of their ideas will perform best (to optimize the customer experience), and to mitigate risk in case the idea goes south. AB Tasty was able to provide this capability!
“When we have a goal to achieve and we have several possible propositions to achieve the same goal, AB Tasty helps us by allowing us to test these propositions. We can then make the best decision by determining which proposition has the greatest impact,” explains Sunny.
One great example is OUI.sncf’s ‘Exploration Assistant’.
Think of the ‘Exploration Assistant’ as an AI-powered travel agent. Based on an in-house algorithm, the Assistant is displayed to visitors with a high enough ‘interest score’, and suggests a trip itinerary for the visitor. The idea sounds great on paper, but would the Exploration Assistant actually make a business impact on bookings?
Using AB Tasty to run a statistically rigorous A/B test, the team determined that conversion rates did significantly increase for those visitors that engaged with the Assistant.
“This test showed extremely positive results: +61% for web responsive and +33% for desktop impact on the conversion rate,” explains Sunny. This particular experiment was important for more than its impact on these KPIs: “This test was a milestone project for Oui.sncf because it was a very technically complex A/B test and was the first test that incorporated AI algorithms. But the work doesn’t stop here. It’s really essential for the teams to continuously iterate on this version of the algorithm, to test and tweak it for better and better results, so that we’re always proposing the most efficient itinerary to our customers. This is how algorithms evolve, and it goes hand in hand with a continuous optimization strategy.”
In other words: targeted, data-driven testing helps validate strategic decisions at key moments, confirming high-level strategy choices and setting future roadmaps.
“Thanks to the Exploration Assistant A/B test and its very impactful results, we will now industrialize this test. This test was very strategic because it was the first which included AI, which plays a key role in OUI.sncf’s strategy.”
– Sunny Song, Lead CRO at OUI.sncf
Another advantage to attaching numbered results to testing ideas? Avoiding circular, time-sapping discussions about whose idea to run with. “By implementing A/B tests and providing concrete results and data to prove the effectiveness of certain ideas, we were able to eliminate the “endless” debates we could have where each person has his own opinion on topics,” concludes Sunny.
For example, low click-through rate data indicated to the team that unfolding travel details on the ticket information page on their app could increase clicks to a key CTA.
This idea could have led to hours of endless debate around the respective value of each of these blocks.
Instead, the team usedFlagship (AB Tasty’s server-side platform) to run an A/B test on the app. This allowed them to quickly validate the winning variation – which happened to be the version B, where the details of the first trip were automatically unfolded. This led to a 2.49% increase in the click-through rate to the CTA “Choose this trip”, as well as 0.28% more arrivals to the basket page.
Flagship: The platform behind the magic
“Our strategy is to offer the most fluid and seamless user experience possible to our clients, the vast majority of whom browse and book on our OUI.sncf mobile app, which has become extremely popular.”
– Ingrid Peiniau Head of Customer Experience at OUI.sncf
Customer experience has become increasingly mobile-first. Alongside buying their groceries, clothing and movie tickets on their smartphones, consumers expect to be able to search, book, or reschedule travel via a user-friendly app. This is especially true for OUI.sncf, where a majority of visits are generated via their mobile app.
This meant that it made sense for OUI.sncf to turn to AB Tasty’s server-side solution, Flagship, for their optimization efforts. It’s the platform they used for tests run on their mobile app, versus their website.
Not only does Flagship enable the kind of experiments OUI.sncf was running on their mobile app, it also allows product teams to release new features with virtually no risk. Why? Teams are able to not only rollout a new feature progressively – that is, not exposing all of their user base at once – but they can also easily rollback that feature if they notice a dip in KPIs.
“We’re particularly interested in Flagship’s progressive rollout capabilities. We want to be done with publication planning dependencies. Progressive rollout lets us stop or fully rollout a test as soon as it reaches statistical significance without having to wait for a new publication of the app. If the chosen KPIs worsen, we can avoid prolonging a sub-optimal experience. On the other hand, if the test is positive, all of our clients can start feeling the benefits immediately.”
From increasing conversion rates on their mobile app to validating high-level strategy and reducing the risk inherent in feature rollouts, Flagship has become a strategic partner for OUI.sncf.
Spark your curiosity!
Get your copy of "OUI.sncf Relies on AB Tasty for Risk-Free Innovation" now.
