For Hélène Doré, brand loyalty is her bread and butter.
As a Conversion Rate Optimization Manager at OUI.sncf, her day-to-day consists of thinking up ways to make customer experiences smoother and conversions higher. It’s a job she loves, but it’s not without its challenges. Today’s consumers are fickle and their expectations are exacting; one poor experience on an app or website can create a long-lasting negative feeling, potentially spiralling into poor brand perception, a drop off to a competitor, or bad word of mouth.
“More likely than not, when a customer faces a problem during his/her visit, it creates a bad memory, which may mean the customer doesn’t return. Creating an exceptional, personalized, and fluid customer experience gives a sentiment of recognition to the customer, which will incite him/her to complete a transaction and potentially come back. If we work on creating the most seamless and personalized customer journey, over time, this recognition will transform into loyalty and create stickiness, inciting the customer to come back and therefore form a virtuous relationship with us.”
– Hélène Doré, Conversion Rate Optimization Manager at OUI.sncf
The conversion rate optimization team, along with the product feature teams, at OUI.sncf has plenty of ideas for how to avoid these scenarios and create truly exceptional customer experiences. The question is: which ideas should they implement? Which should they scale? And how can they be sure an idea won’t backfire?
A/B Testing: The way of data, not debates
What Hélène and her manager Sunny Song, Lead CRO, needed was the ability to run data-driven experiments in order to know which of their ideas will perform best (to optimize the customer experience), and to mitigate risk in case the idea goes south. AB Tasty was able to provide this capability!
“When we have a goal to achieve and we have several possible propositions to achieve the same goal, AB Tasty helps us by allowing us to test these propositions. We can then make the best decision by determining which proposition has the greatest impact,” explains Sunny.
One great example is OUI.sncf’s ‘Exploration Assistant’.
Think of the ‘Exploration Assistant’ as an AI-powered travel agent. Based on an in-house algorithm, the Assistant is displayed to visitors with a high enough ‘interest score’, and suggests a trip itinerary for the visitor. The idea sounds great on paper, but would the Exploration Assistant actually make a business impact on bookings?
Using AB Tasty to run a statistically rigorous A/B test, the team determined that conversion rates did significantly increase for those visitors that engaged with the Assistant.
“This test showed extremely positive results: +61% for web responsive and +33% for desktop impact on the conversion rate,” explains Sunny. This particular experiment was important for more than its impact on these KPIs: “This test was a milestone project for Oui.sncf because it was a very technically complex A/B test and was the first test that incorporated AI algorithms. But the work doesn’t stop here. It’s really essential for the teams to continuously iterate on this version of the algorithm, to test and tweak it for better and better results, so that we’re always proposing the most efficient itinerary to our customers. This is how algorithms evolve, and it goes hand in hand with a continuous optimization strategy.”
In other words: targeted, data-driven testing helps validate strategic decisions at key moments, confirming high-level strategy choices and setting future roadmaps.
“Thanks to the Exploration Assistant A/B test and its very impactful results, we will now industrialize this test. This test was very strategic because it was the first which included AI, which plays a key role in OUI.sncf’s strategy.”
– Sunny Song, Lead CRO at OUI.sncf
Another advantage to attaching numbered results to testing ideas? Avoiding circular, time-sapping discussions about whose idea to run with. “By implementing A/B tests and providing concrete results and data to prove the effectiveness of certain ideas, we were able to eliminate the “endless” debates we could have where each person has his own opinion on topics,” concludes Sunny.
For example, low click-through rate data indicated to the team that unfolding travel details on the ticket information page on their app could increase clicks to a key CTA.
This idea could have led to hours of endless debate around the respective value of each of these blocks.
Instead, the team used Flagship (AB Tasty’s server-side platform) to run an A/B test on the app. This allowed them to quickly validate the winning variation – which happened to be the version B, where the details of the first trip were automatically unfolded. This led to a 2.49% increase in the click-through rate to the CTA “Choose this trip”, as well as 0.28% more arrivals to the basket page.
Flagship: The platform behind the magic
“Our strategy is to offer the most fluid and seamless user experience possible to our clients, the vast majority of whom browse and book on our OUI.sncf mobile app, which has become extremely popular.”
– Ingrid Peiniau Head of Customer Experience at OUI.sncf
Customer experience has become increasingly mobile-first. Alongside buying their groceries, clothing and movie tickets on their smartphones, consumers expect to be able to search, book, or reschedule travel via a user-friendly app. This is especially true for OUI.sncf, where a majority of visits are generated via their mobile app.
This meant that it made sense for OUI.sncf to turn to AB Tasty’s server-side solution, Flagship, for their optimization efforts. It’s the platform they used for tests run on their mobile app, versus their website.
Not only does Flagship enable the kind of experiments OUI.sncf was running on their mobile app, it also allows product teams to release new features with virtually no risk. Why? Teams are able to not only rollout a new feature progressively – that is, not exposing all of their user base at once – but they can also easily rollback that feature if they notice a dip in KPIs.
Benoit Bouffart, Chief Tech & Product Officer at e.Voyageurs SNCF., explains:
“We’re particularly interested in Flagship’s progressive rollout capabilities. We want to be done with publication planning dependencies. Progressive rollout lets us stop or fully rollout a test as soon as it reaches statistical significance without having to wait for a new publication of the app. If the chosen KPIs worsen, we can avoid prolonging a sub-optimal experience. On the other hand, if the test is positive, all of our clients can start feeling the benefits immediately.”
From increasing conversion rates on their mobile app to validating high-level strategy and reducing the risk inherent in feature rollouts, Flagship has become a strategic partner for OUI.sncf.
