Rollouts Archives

For Hélène Doré, brand loyalty is her bread and butter.

As a Conversion Rate Optimization Manager at OUI.sncf, her day-to-day consists of thinking up ways to make customer experiences smoother and conversions higher. It’s a job she loves, but it’s not without its challenges. Today’s consumers are fickle and their expectations are exacting; one poor experience on an app or website can create a long-lasting negative feeling, potentially spiralling into poor brand perception, a drop off to a competitor, or bad word of mouth.

“More likely than not, when a customer faces a problem during his/her visit, it creates a bad memory, which may mean the customer doesn’t return. Creating an exceptional, personalized, and fluid customer experience gives a sentiment of recognition to the customer, which will incite him/her to complete a transaction and potentially come back. If we work on creating the most seamless and personalized customer journey, over time, this recognition will transform into loyalty and create stickiness, inciting the customer to come back and therefore form a virtuous relationship with us.”

– Hélène Doré, Conversion Rate Optimization Manager at OUI.sncf

The conversion rate optimization team, along with the product feature teams, at OUI.sncf has plenty of ideas for how to avoid these scenarios and create truly exceptional customer experiences. The question is: which ideas should they implement? Which should they scale? And how can they be sure an idea won’t backfire?

A/B Testing: The way of data, not debates

What Hélène and her manager Sunny Song, Lead CRO, needed was the ability to run data-driven experiments in order to know which of their ideas will perform best (to optimize the customer experience), and to mitigate risk in case the idea goes south. AB Tasty was able to provide this capability!

“When we have a goal to achieve and we have several possible propositions to achieve the same goal, AB Tasty helps us by allowing us to test these propositions. We can then make the best decision by determining which proposition has the greatest impact,” explains Sunny.

One great example is OUI.sncf’s ‘Exploration Assistant’.

Think of the ‘Exploration Assistant’ as an AI-powered travel agent. Based on an in-house algorithm, the Assistant is displayed to visitors with a high enough ‘interest score’, and suggests a trip itinerary for the visitor. The idea sounds great on paper, but would the Exploration Assistant actually make a business impact on bookings?

Using AB Tasty to run a statistically rigorous A/B test, the team determined that conversion rates did significantly increase for those visitors that engaged with the Assistant.

“This test showed extremely positive results: +61% for web responsive and +33% for desktop impact on the conversion rate,” explains Sunny. This particular experiment was important for more than its impact on these KPIs: “This test was a milestone project for Oui.sncf because it was a very technically complex A/B test and was the first test that incorporated AI algorithms. But the work doesn’t stop here. It’s really essential for the teams to continuously iterate on this version of the algorithm, to test and tweak it for better and better results, so that we’re always proposing the most efficient itinerary to our customers. This is how algorithms evolve, and it goes hand in hand with a continuous optimization strategy.”

In other words: targeted, data-driven testing helps validate strategic decisions at key moments, confirming high-level strategy choices and setting future roadmaps.

“Thanks to the Exploration Assistant A/B test and its very impactful results, we will now industrialize this test. This test was very strategic because it was the first which included AI, which plays a key role in OUI.sncf’s strategy.”

– Sunny Song, Lead CRO at OUI.sncf

Another advantage to attaching numbered results to testing ideas? Avoiding circular, time-sapping discussions about whose idea to run with. “By implementing A/B tests and providing concrete results and data to prove the effectiveness of certain ideas, we were able to eliminate the “endless” debates we could have where each person has his own opinion on topics,” concludes Sunny.

For example, low click-through rate data indicated to the team that unfolding travel details on the ticket information page on their app could increase clicks to a key CTA.

This idea could have led to hours of endless debate around the respective value of each of these blocks.

Instead, the team used Flagship (AB Tasty’s server-side platform) to run an A/B test on the app. This allowed them to quickly validate the winning variation – which happened to be the version B, where the details of the first trip were automatically unfolded. This led to a 2.49% increase in the click-through rate to the CTA “Choose this trip”, as well as 0.28% more arrivals to the basket page.

Flagship: The platform behind the magic

“Our strategy is to offer the most fluid and seamless user experience possible to our clients, the vast majority of whom browse and book on our OUI.sncf mobile app, which has become extremely popular.”

– Ingrid Peiniau Head of Customer Experience at OUI.sncf

Customer experience has become increasingly mobile-first. Alongside buying their groceries, clothing and movie tickets on their smartphones, consumers expect to be able to search, book, or reschedule travel via a user-friendly app. This is especially true for OUI.sncf, where a majority of visits are generated via their mobile app.

This meant that it made sense for OUI.sncf to turn to AB Tasty’s server-side solution, Flagship, for their optimization efforts. It’s the platform they used for tests run on their mobile app, versus their website.

Not only does Flagship enable the kind of experiments OUI.sncf was running on their mobile app, it also allows product teams to release new features with virtually no risk. Why? Teams are able to not only rollout a new feature progressively – that is, not exposing all of their user base at once – but they can also easily rollback that feature if they notice a dip in KPIs.

Benoit Bouffart, Chief Tech & Product Officer at e.Voyageurs SNCF., explains:

“We’re particularly interested in Flagship’s progressive rollout capabilities. We want to be done with publication planning dependencies. Progressive rollout lets us stop or fully rollout a test as soon as it reaches statistical significance without having to wait for a new publication of the app. If the chosen KPIs worsen, we can avoid prolonging a sub-optimal experience. On the other hand, if the test is positive, all of our clients can start feeling the benefits immediately.”

From increasing conversion rates on their mobile app to validating high-level strategy and reducing the risk inherent in feature rollouts, Flagship has become a strategic partner for OUI.sncf.

Spark
your curiosity!

Get your copy of "OUI.sncf Relies on AB Tasty for Risk-Free Innovation" now.

Weekendesk shares a sneak peek how AB Tasty’s Roll Outs solution allow them achieve faster releases with less risk

Travel website Weekendesk has a simple mission: help hard-working weekend warriors easily find the best deals for their trips. They were also one of the first early adopters of our feature management platform for product teams. Their tech and product teams got a sneak peek of the tool, and got to play around before the official release. Here’s what they thought:

To start off – what got you interested in AB Tasty?

Generally speaking, what are the advantages of using a feature management and product experimentation platform?

Julien Martin, VP of Engineering: A/B testing is essential for any business. But in our case, I was very particular about the kind of tools I wanted to use. With our new front-end application, I decided to block any ‘classic’ (client-side) A/B testing solution – I didn’t want a layer of JavaScript/CSS tag on there, to ensure maximum performance.

We thought about this quite a bit, and after trying our own (pretty limited) in-house system, AB Tasty told us about their Feature Experimentation and Roll Outs functionality. It was the solution to all of our issues – no added JS/CSS layer, and no development outside our repository, which will let us maintain the same high level of quality and performance in any situation.

Alexandre Duarte and Alessandro Pintaudi, Co-Heads of Product: The platform lets us be more autonomous and also mitigates risk thanks to progressive rollouts. We can test new features on specific segments of our user base, and also better integrate user feedback in real-time, set up kill switches, and especially, run experiments.

What were your first impressions of the platform?

Maxime Biloé, Lead Front-End Developer, and Sonia Alliche, Front-End Developer: We think the interface was really clear, everything is very logical and easy to understand, even with the variety of options available in the platform. We were pleasantly surprised at how easy and quick it was to set up a test, from the interface itself to the integration with our application.

Alexandre and Alessandro: The dashboard is easy to use, especially since there are pre-defined use cases – this is a real time-saver. The interface is also very intuitive and easy to get started with the first time around. This is true in terms of configuring the tests, retrieving data and analyzing the reports. Finally, the support we received from AB Tasty allowed us to deploy the tool very quickly, and to launch our first tests in only a few days.

Flagship platform interface

What challenges were you able to tackle thanks to our platform?

Julien: We started with a really simple test, just to get used to the platform. From a technical perspective, the idea was to put the performance of the tool to the test. Just one extra https request to create a better user experience and better processes – this is just perfect.

Alexandre and Alessandro: We want to be quicker and more efficient in our releases, without increasing any risks associated with putting new features live. The platform lets us do this thanks to the different functionalities available in the platform (progressive rollout, A/B testing, feature flagging…), as well as advanced experimentation to optimize our product.

After playing around with the tool, the major takeaway for us is that we need to systematize and speed up how we deploy A/B tests, so that we can better understand the issues our user base faces, and find solutions – all with less risk.

What are the next steps for Weekendesk and AB Tasty?

Julien: We’re going to integrate a new payment service provider (PSP) in our application. The platform lets us deploy a progressive rollout, so we can avoid any messy roll-backs and confirm our theories. We’re organized into squads, who will be working on continuous optimization on a variety of subjects, and therefore running a lot of tests at once.

Alexandre and Alessandro: We’re going to run a series of A/B test campaigns at the same time during the last stages of the purchase funnel. One of our goals here is to simplify the decision to buy something, but also to uncover the factors that lead up to it.