Author: AB Tasty

AB Tasty is a complete tool for website and conversion rate optimization. We serve as your digital lab, equipped with everything you need to create experiments that will help you to better understand your users and customer journeys so that you can create the clearest and most engaging user experience possible, ensuring your website performs well and yields the maximum results.

Article

Apr 12, 2022
9min read

The Rise of the Experience Economy and How It’s Shaping the Future of E-Commerce

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

What is the experience economy?

Let’s say you have an online shop and in that online shop you have a product. Your product is designer eyewear and prescription glasses. A customer visits your online shop to learn about your product. That customer needs to determine which frames will suit their face and what size to order. A similar shop that sells similar products to yours offers free shipping and free returns of up to 3 pairs at no charge, or the use of a virtual reality assistant, via their mobile app, to help their customers make purchasing decisions without needing to visit a store. Your shop, though well-intentioned and bug-free, does not. The customer’s experience researching and selecting their product is what ultimately drives their decision-making process, and they purchase from the other shop. And the next time they need glasses, they purchase from that other shop again. That’s the experience economy.

What is the experience economy

In the experience economy, finding a differentiating edge is crucial for brands (Source)

Expressed in more academic terms, the experience economy is the packaging of goods and services into a bundle such that the experience of acquiring or consuming is the key selling point – it’s the reason the customer came into your shop in the first place.

In 1998, two Harvard researchers published an article detailing the concept of the experience economy for the first time, using a birthday cake analogy to eventually draw out the definition we see above. These days, the concept is more important than ever, as the rapidly evolving digital transformation of the way we consume information and goods creates a never-ending, multi-channel interaction between brands and consumers. And it’s key to your overall business success.

How e-commerce brands can succeed in the experience economy

In the age of digitalization, not only do all brands have websites, incorporating an e-commerce platform for online sales, but they also have Facebook, Instagram, TikTok and Snapchat accounts, more than likely a YouTube channel, a web browser adapted to mobile devices and an app to sit alongside it. In short, multiple channels and touchpoints for their customers to interact and engage with them, and multiple opportunities to create experiences to acquire new customers and drive sales. This all makes for a non-linear shopping experience, and requires careful examination of what customers expect on which channel and at which time.

How the experience economy is shaping the future

A customer-first mindset is crucial for businesses that are looking to win the digital CX game (Source)

How can brands adapt to shifting consumer preferences

At AB Tasty, we’re convinced that the brands opting for a “business as usual” approach will quickly be left in the dust. Customers expect better servicing, more meaningful interactions and suggest that they’ll spend more when brands deliver. This means having a strategy that considers multiple channels, across physical, digital and social touch points, and adapts to the preferences of each individual so that interactions remain authentic and personal. If you’re engaging with customers without being able to have in-person contact, experience matters even more, because consumers still want to be seen as individuals with their own unique needs. Ultimately, their experience will influence their buying decisions and according to Salesforce, 66% of consumers expect companies to understand their unique needs and preferences.

The shift in brand-consumer interactions

Create a personalized, relevant shopping experience for each customer (Source)

Figuring out what your customers want doesn’t just need to be a guessing game, experimentation is standard practice for the experience economy. In B2C environments, marketing teams test website performance using a range of experiments that examine layout, colors, purchase journeys, product information and visual features to ensure no stone is left unturned in maximizing transactions and revenue. And adopting an experimentation mindset really is a win-win. On the one hand, you’re identifying the best way to interact with your customers – identifying what they respond to and what they want – and on the other, you’re maximizing every opportunity to drive purchases and serve your bottom line.

Why prioritizing customer experiences matters

That’s all very well and good, you might say, but what difference does it really make? Plenty, in fact. Relevant and personalized consumer experiences are key to keeping your brand ahead of its competitors. Let’s explore some of the reasons for this.

  1. Loyalty is hard-earned and easily lost
    PWC’s 2021 Global Consumer Insights Survey found that 84% of shoppers trust brands that provide exceptional customer service, but one in three will walk away after just one negative shopping experience. In a similar vein, Qualtrics’ 2022 Global Consumer Trends survey reported that 60% of consumers would buy more if businesses treated them better, and also determined that 9.5% of your overall revenue is at risk from negative shopping experiences. These statistics still haven’t convinced you? Read on!
    ă…€
  1. Seamlessness is synonymous with success
    You can design any number of gimmicks to attract attention, but it’s the seamless ones that stick. Take the Clarins Singles Day Wheel of Fortune promotion, where any customer landing on the brand’s desktop or mobile site in EMEA saw a pop-up to spin the wheel. They were then rewarded with one of six special offers, which was automatically added to their basket via a promo code at the checkout. This automatic add proved crucial: Results were strong across all key territories, with Ireland particularly notable, seeing a 495% increase in orders and a 585% increase in revenue. Clarins uncovered a clever, engaging offer and coupled it with a seamless UX process for their shoppers, delivering simply stunning results.Clarins case studyClarins delivered a customer experience on par with their clients’ expectations (Source)
    ă…€
  2. Stagnate and you’ll be left behind
    To innovate or not to innovate, is it even a question? If you’re thinking about it, then your competitors almost certainly are too. And if you’re not trying something new, you almost certainly risk falling behind. While bug-free websites and a smooth journey through the purchase funnel is great, it’s also the bare minimum that you should be doing. Salesforce found that 91% of customers are more likely to make a repeat purchase from a company after a positive customer experience. Delivering a seamless, multichannel experience across all business interactions is integral to staying ahead and it’s clear there is still scope for brands to optimize.

4 examples of brands that are excelling in the experience economy

As we’ve seen in the above section, brands that embrace the experience economy are best-positioned to see increased loyalty, repeat business, and convert their customers into advocates for their products. Pushing beyond experiences into memorable interactions for their consumers has allowed some of the best-known brands in the world to gain further ground on their direct competitors, all while staying true to their core values. Let’s take a look at the best-in-class trends and examples of the experience economy model.

Nike

Nike is driven by delivering innovative products, services and experiences to inspire athletes. One such experience is their Nike Fit solution: an AI-driven app that allows you to virtually measure and fit your foot to ensure you choose the right pair of Nike shoes, no matter the style nor the shape of your foot, and without having to leave your living room.

Nike Fit

Nike introduces innovative solutions to their clients’ biggest point of friction (Source)

Sephora

In 2019, Sephora pioneered their intelligent digital mirror in the brand’s Madrid flagship, using the power of AI to deliver hyper-personalized experiences and product recommendations to shoppers. The mirror not only allows consumers to “test” products by displaying how they’ll render when applied, it also provides personalized product recommendations and suggestions based on an analysis of the customer’s features.

Sephora

Sephora develops new ways to offer their customers personalized recommendations (Source)

Starbucks

Starbucks has revolutionized their physical footprint by opening pickup-only stores in key, high-traffic locations where rental space is at a premium and busy lives mean in-and-out transactions are the order of the day. This store concept allows coffee lovers to order and pay ahead of time, via the Starbucks mobile app, and nominate the pickup location, for a speedy service that saves tedious, peak-hour queues. Not to mention a boost to sales per square foot, a key metric in the brick-and-mortar retail space.

Starbucks

Starbucks identifies their customers’ needs and delivers an optimal shopping experience (Source)

Asos

This online fashion retailer was founded in London in 2000, and now sells over 850 brands around the world. In identifying one of the key barriers to online shopping for clothes – choosing the correct size – Asos developed their Fit Assistant tool to ensure customers could navigate the online shopping experience hassle-free. Available on both desktop and mobile, Fit Assistant delivers personalized recommendations according to shoppers’ individual shapes and sizes.

Fit Assistant Technology - Recommend

Asos optimizes their customers’ online shopping experience (Source)

Why the experience economy is here to stay

Through a combination of rapid digital transformation, technological innovation of smart devices (phones, tablets, watches and more), and the increasing pace of our daily lives, the manner in which we consume products has evolved beyond mere acquisition. How we consume the product matters. How we feel about how we consume the product matters. How the brand ensures we enjoy our consumption of the product matters. And if your brand is not up for the challenge and staying ahead of the game, consumers will find one that is. It’s as simple as that. Evolve, innovate, and deliver seamless brand experiences, and you’ll lead the competition, win market share and generate growth.

If you’re looking for some guidance on how to deliver impactful brand experiences that will “wow” your customers, draw inspiration from the first-ever digital customer journey that maps out how to drive optimization and innovation to take your customer experience to the next level.

Share on

LinkedIn Facebook Twitter

Article

Mar 1, 2022
14min read

Everything You’ve Ever Wanted to Know About Landing Page Design

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

You’ve spent months building your brand and getting your website just right, but it’s all for nothing if you can’t turn your hard work into sales. And landing pages are one of the best resources you can use to optimize your conversions. Well-designed landing pages can lead customers to specific products or services, encourage them to take immediate action and capture contact details to build your customer base.

Landing pages are highly effective conversion tools
Landing pages are highly effective conversion tools (Source)

Landing pages have conversion rates ranging from 3% to 11.45% and higher, depending on the features they include. How well your page performs depends entirely on how well you’ve designed it and running tests on your page can help you continuously find the winning formula.

What is a landing page?

Landing pages are website pages designed to target a specific visitor according to a particular demographic, interest, or buying behavior. Your landing page should attract leads in the most effective way possible and then convert your visitors into buyers or nudge them towards taking another kind of action.

Landing pages are designed to encourage exploration and could have a number of goals. Technically, any page on your website can become a landing page for a campaign, but this could distract your customers and send them down a rabbit hole instead of focusing on the action you want them to take. Good landing pages have a singular focus or call to action (CTA), making them excellent marketing and lead generation tools.

Why is landing page design so important?

Landing pages let you nudge your leads straight towards a conversion through strategically placed call to action elements like links, forms, buttons, and others. A well-designed landing page lures visitors through a message that piques their interest, like a discount, a piece of content they’d like to read, or a free trial to test a product.

Continuously testing your pages
Landing pages target customers based on specific demographics, like browsing behavior (Source)

While it doesn’t fully show off your brand’s personality, it does lay the groundwork for getting to know it better. When a customer clicks on a call to action, they want to find more information about whatever has caught their eye, and that’s when you’ll convert them from a visitor to a solid lead or even a paying customer.

Aside from optimizing your users’ actions, landing pages make it easy to track visitors and modify your pages as required. Through A/B testing, one item on your landing page is changed on the variation to determine the impact of the change.

The importance of lead capture landing pages

Lead capture landing pages (sometimes called squeeze pages) can also gather important customer details (such as names, email, phone numbers). Lead capture landing pages offer your customers something of interest (e.g., a free trial or e-book) in exchange for completing a short form that collects their personal information.

You can use these personal details to target them in the future, with email campaigns, social media advertising, or follow-up calls by your sales representatives. Targeting potential customers strengthens the likelihood of a conversion down the line.

The essential elements of a landing page

A landing page should always have one conversion goal. From headlines to images to buttons, every element should bring you closer to that objective.

Clear and effective copywriting

An excellent landing page needs punchy, clear copywriting to convert. You should describe the benefits of your product/service and what your company does as concisely as possible. Don’t overstuff your landing page with copy that doesn’t add value!

A few copywriting tips to bear in mind:

  • Always direct your visitors to the primary call to action with your copy.
  • Keep sentences short and to the point.
  • Avoid using too many adverbs (quickly, obviously, actually).
  • Instead of passive voice, use active voice to inspire action.
  • Edit your work to delete anything that adds more detail than necessary.

Visually attractive and clear CTAs

Your call to action (CTA) is the most important element on your landing page. When you create your CTA, use copy that inspires the visitor to continue along their journey with you. Compare generic calls to action like “Click Here” or “Submit” to powerful statements like “Yes, I want to save money” or “Update my wardrobe.” The latter sounds better!

Always make sure that your CTA button stands out from the other elements on the page. Follow the example set by the Dutch watch brand Cluse: When their team noticed the bounce rate for their landing pages was high and that not many users advanced to the product display pages, it became evident that the CTA on the page wasn’t clear enough.

By changing the call to action used in their landing page design to best match practice guidelines, they instantly saw an uplift in conversions and sales. The winning variation increased the click-through rate to their product display page by 2.39% and realized a 1.12% uplift in transactions.

Eye-catching headers

Good landing pages have simple, compelling and uncluttered headers. If you include an image, make sure it’s related to your product or service. Keep all the action above the fold and use directional cues to steer visitors to that all-important CTA button. Additionally, you should avoid including distracting elements like links or phone numbers.

Social proof

People will conform to the majority in order to be accepted or liked. This is known as social proof. If a prospect sees that another person likes a product or has had a positive experience with your company, your odds of conversion go up.

When you’re browsing a landing page and see a testimonial from an industry expert you respect, that’s social proof. When you’re cruising a pricing page and you see that an industry giant is already using the tool, that’s social proof. When you sign up for a demo because you know the tool solved the exact problem you have for a similar company, that’s social proof.

Think of it as borrowing third-party influence to sway potential customers. Brightlocal found that the average consumer reads at least ten reviews before trusting a business. Your odds of converting goes up when you see that others had used the product before them and were happy with the end result.

Social proof may include customer reviews, a list of existing customers, user testimonials and awards you may have earned. Take a page from Decathlon’s playbook: By testing different options, you can also identify which works best for winning over your customers!

A unique selling proposition (USP)

What makes you different from everyone else? You don’t need to be more advanced or offer huge discounts; you just need a convincing brand promise. Are you the fastest? Most reliable? Most knowledgeable? You only have a few lines of copy and limited visuals to get your point across, so be short and sharp.

Hero images

First impressions are essential — and when it comes to website optimization, they’re more important than ever! A hero image is a large banner that appears at the top of your page, usually occupying the full length of the screen. Look for an image that represents (or is relevant) to your product or company. You can even include a call to action and/or a text overlay to guide your users down the conversion funnel.

Additional information in the footer

Remember that even though you never want to distract customers from the call to action, you’ll still want them to find more information about your company if they don’t convert immediately.

Keep your calls to action and more compelling arguments above the fold, but include additional information like newsletter sign-ups, links to your social media pages, or ‘About us’ pages in the footer for customers that need to know a little more about you before proceeding.

Landing pages should be simple and focus on the call to action
Landing pages should be simple and focus on the call to action (Source)

How to design and optimize your landing pages

Now that you understand some of the best practices for designing high-converting landing pages, let’s dig into the nitty-gritty of your landing pages’ design. Here are ten best practices to design a landing page that converts successfully:

1. Identify your target audience and their needs

Designing a visually stunning landing page doesn’t necessarily mean it’s effective. Truly great landing pages understand what visitors are looking for and provide quality content tailored to your target audience. Before designing your page, establish who it’s for and what they would like to know. Keep testing variations of the page to see which elements resonate the most with your audience and your landing page can become a powerful conversion tool.

2. Use strategic and relevant hero images

There should be a clear connection between your brand and the hero image you display on your landing page. Misplaced images can confuse visitors or create a poor impression of the brand. Make sure that the image you use fosters an instant connection between you and your users by either answering a question, nudging them to advance down the conversion funnel, or just encouraging them to find out more about you.

3. Refine your headlines

We’ve already spoken about the role copywriting plays in conversion, so make sure that you continually test and refine your headlines. Landing page headlines promote your value proposition, while the subheadings provide further explanations. Keep these short, sharp and to the point. If you aren’t sure which value proposition to promote, try testing different versions to see which version resonates the most with visitors and results in the most click-throughs.

4. Keep navigation simple

Landing pages have to be optimized to create the best user experience possible. Keeping navigation simple, intuitive and straightforward will boost user retention and lead generation while driving your core message home. ECCO Shoes are a great example of this: By making their call to action button larger and repositioning it above the fold, cutting a few visual distractions and adding icons highlighting special offers like free delivery, their new landing page was able to outperform the original by over 17%!

5. Harmonize your colors

Colors can evoke powerful emotions or cause confusion. Use color strategically and sparingly on your landing page so it highlights the most important elements like your call to action, add-to-cart button or contact forms. You can also use negative space to break up the page and make it easier to digest the most important information.

6. Place your important elements above the fold

The ‘above the fold’ section is the part of your website that’s visible in the browser without scrolling down. It’s the first thing people see, so make sure you grab their attention with compelling imagery, copy and headers.

7. Make sure it displays properly across devices

Today’s consumers are using a host of different devices and operating systems. This means your website has to display properly across all of them if you want their business. Ensure that your landing pages are optimized to display correctly on all available platforms.

8. Optimize your site’s load times

Your site’s loading time affects your user experience and your Google ranking. Slow websites contribute to high bounce rates and cart abandonment. You can optimize your load times by enabling compression, using web-friendly imagery and keeping redirects to a minimum.

9. Improve your SEO to increase traffic and conversions

SEO (Search Engine Optimization) is a complex subject, but anyone can make small tweaks to their landing page to improve their ranking. Ensure that your title tags and meta-descriptions are accurate and relevant to what visitors will find on your page. Always include the main keywords you want to rank for, but don’t overstuff your page with them; it will negatively affect your user experience and Google may penalize you for it.

Effective landing pages can lead to increased sales (Source)

10. Perform testing on your landing pages

A/B testing allows you to compare two versions of the same page to see which version resonates the most with your users. By changing an element on one page, you can compare the two to see which changes will increase conversions. You can learn more about testing on your landing page in this article, or try our A/B testing tool.

A/B testing has numerous benefits for your landing pages and your business because it reveals incredible insights that can be used to:

  • Improve user engagement by learning which messages/images resonate with customers;
  • Present more relevant messages and content that customers actively respond to;
  • Reduce bounce rates by improving relevance;
  • Increase conversion rates;
  • Deliver quick results and improvements that provide a return on investment.

Common landing page mistakes

We’ve covered what you should do when you are designing your landing page, but there are also a few things you should avoid doing:

  • Multiple CTAs: Too many calls to action dilute your message and decrease the likelihood of performing a conversion.
  • Broken CTAs: If your call to action doesn’t link to the right page or function properly, all of your efforts are wasted.
  • Low-resolution images/no images: Copy alone won’t sell your product or service. Use high-resolution, web-optimized images to boost your brand and convey your message visually.
  • Poorly written copy: Spelling mistakes and grammatical errors impact the trust that consumers have in your brand.
  • Lengthy forms: If your lead capture forms are too long, your customers will get frustrated and may give up halfway through. You’ll lose the information — and their business.

Examples of great landing page design

While landing page design is an ongoing process and you need to keep constantly optimizing it, there are a few brands with really great landing pages to inspire your own.

Coinbase

Coinbase example

Coinbase uses a stylish and focused lead generation page with a clear message and unique value proposition. While trading cryptocurrency is a complex topic, the landing page simply asks you to enter your email address to get started. This example works because the call to action is clear, the copy is straightforward and the design is uncluttered.

Xero

Xero example

Xero uses negative space and compelling copy to get its message across. There are two call to action buttons (with the same message) to steer visitors in the right direction. Xero’s page is attractive and colorful, uncluttered and emphasizes the call to action button, making it highly effective.

monday.com

Monday example

Monday.com uses colorful buttons to encourage visitors to create their free account. They’ve also added compelling social proof messages by mentioning the number of visitors to the site. This use of social proof and a clear, uncluttered layout — coupled with direct, pointed copy — increases their landing page’s ability to convert.

Astra

Astra example

Astra uses compelling copy in their click-through button that assures users that they can secure your website against hackers in just three minutes. They’ve also added some big-name logos in their footer to build trust in the brand. The tagline, ‘Are you next?’ is compelling, with a solid and distinct call to action leading prospects to the sign-up page.

Take your landing pages to the next level

These tips provide a great starting point for designing a high-converting landing page, but no one gets it right the first time. Building an effective landing page requires ongoing optimization and innovation to determine what resonates with your audience.

Continuously testing your pages leads to new insights and continual improvements
Continuously testing your pages leads to new insights and continual improvements (Source)

Experimentation can help your brand improve the performance of your landing pages. Continuously testing your pages leads to new insights and improvements that’ll help you constantly adapt to new customer preferences even as they are being formed.

Looking for inspiration to take your landing pages to the next level? Check out our e-book, “50 Tests You Should Know For Website Optimization”: It contains 50 successful experiments from e-commerce businesses who have continuously optimized their websites to offer relevant and personalized experiences.

Share on

LinkedIn Facebook Twitter

Article

Jan 26, 2022
3min read

Introducing AB Tasty’s New Navigation

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

At AB Tasty, we love to help you improve your customers’ experiences – and we are here to do the same for you on the AB Tasty platform! We’re constantly gathering feedback from our users, and next month, you’ll see us roll out our new navigation based on that feedback.

We’re doing this for a few reasons:

  1. We want to give you the best – and that means further improving the quality of your experience on the platform. ?
  2. We want you to be able to find exactly what you need, when you need it – which means improving the organization of information, classifying your favorite (and new!) features in an easy-to-navigate way. ?
  3. We want you to have the most intuitive experience possible – by providing you with better guidance from the first time you log in and get you from A to B as quick as can be. ?

What does that mean for you?

We’ll guide you through the updates in the coming weeks, but here’s a sneak peek of what to expect:

  1. Better visibility with a new sidebar navigation, allowing you to easily access any area of the platform with a single click – and collapse it for more workspace.

    • We’ve gotten rid of the hamburger menu in favor of giving you more control over where you want to go within the platform – whether it be Tests, Personalization, Audience, Analysis, or ROI – plus a login button to take you directly to Flagship, our feature management solution. ?
  2. Improved access to Settings, reorganized to match our customers’ most-used options.
    • We’ve designed a sleeker look, consolidating settings menu for a cleaner appearance and easier navigation. ?
  3. New header to accompany you through every step of the workflow, from campaign creation to reporting, giving you a better bird’s eye view of a campaign’s status.

    • Your step-by-step buttons will remain exactly where they are, but the header will shift to make everything more easily visible to you – including an editable campaign name, status, and reporting, right alongside the tag and account info. ?

We hope these exciting changes make a big impact on how you use AB Tasty! ?

We know you might have questions as you go through the new navigation, and we are here to help! We also know you might have feedback – about the new design and beyond – and we invite you, as always, to share it with us on our Canny board, accessible via this link.

See you on the new navigation soon!

Share on

LinkedIn Facebook Twitter

Article

Jan 11, 2022
5min read

1,000 Experiments Club: A Conversation With Lukas Vermeer of Vista

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

When it comes to kickstarting experimentation within an organization, Lukas Vermeer recommends starting small and (keeping it) simple.

Lukas Vermeer took this advice to heart when he dove head-first into the world of AI and machine learning during the early stages of its development, when there was little industry demand. Through consulting for various companies, Lukas discovered his ideal work environment: a scale-up, where he could put his data and machine learning expertise to use.

Enter Booking.com. Lukas joined the Dutch digital travel company during the scale-up phase and went on to lead the experimentation team for eight years, scaling the team from three people to 30 people.

Once the experimentation team at Booking.com had reached maturity, he embarked on a new adventure in 2021 as director of experimentation at Vista. He is building and shaping the experimentation culture and tapping into the potential of their data, to further Vista’s impact as an industry leader in design and marketing solutions for small businesses.

Lukas spoke with AB Tasty’s VP of Marketing Marylin Montoya about the process and culture of experimentation; from the methods to the roles of the teams involved within an organization. Here are some of the key insights from their conversation.

Get strategic about experimentation

Knowing the purpose of your experiment is key. Lukas recommends focusing your efforts on testing big features that can drive real change or impact the company’s bottom line, rather than UI design.

Ask yourself, “What are the biggest questions that are driving your business case at the moment? What are the biggest assumptions that are behind your strategic planning?” he says. Rather than increasing the number of experiments, focus on the correct execution of more significant experiments.

When it comes to building a culture of experimentation within an organization, Lukas suggests using the flywheel method. The first experiment should garner attention by splitting the company’s opinion 50/50, as to whether it will work. This demonstrates that it can be hard to predict the success of experiments, thereby underlining the “unquantifiable value of experimentation.” We need to acknowledge that it is equally valuable to avoid shipping a bad product (that could reduce revenue), as it is to figure out strategically what you should invest in going forward.

Structure your organization for experimentation success

The way your business and teams are structured will impact how seamlessly your experiments are executed. Lukas recommends that the product development team take full ownership of the experiments.

The experimentation team should be facilitating experiments by providing the tools, education and troubleshooting support to the product development team, who can then run their experiments autonomously.

By training product managers in the process of experimentation — such as the different tests and tools available, their strengths and weaknesses, the assumptions they make and when to use them — they can work autonomously to test their ideas and select from a portfolio of experimental methods in order to make a decision.

There is, however, a social aspect to experimentation that should not be ignored. Given the subjective nature of data interpretation and analysis, Lukas highlights the importance of discussing the outcomes and giving feedback on the experimentation process in order to optimize it.

“The whole point of an experiment is to (…) drive a decision, and the decision should be supported by the evidence at hand,” Lukas says. Just as scientists peer-review their papers before publishing, experiments using the scientific method should follow the same guidelines to document the hypothesis, method, results and discussion in the reporting. (An opinion that has been echoed by 1,000 Experiments Club podcast guest Jonny Longden.)

The biggest threat to experimentation culture: leadership or roadmaps?

When people in product development talk about “roadmaps,” they’re not actually roadmaps, Lukas says. It’s more of a linear wishlist of steps that they hope will bring them to the goal. The problem is that there’s rarely alternative routes or redirections should they stray from the original plan.

It’s hard to change direction at the first failed experiment, Lukas explains, due to the  “escalation of commitment.” That is, the more time and energy you have invested into something, the more difficult it is to change course.

So, is it time to ditch roadmaps altogether? Lukas advises that roadmaps should simply acknowledge that there is inherent uncertainty. There are many unknowns in product development, and these only become visible once the products are being built and exposed to customers. This is why the build-measure-learn model works, because we take a few steps and then check if we’re heading in the right direction.

Lukas says the goal should not be to “deliver a final product in two months,” rather you should incorporate the uncertainty into the deliverables and word the objective accordingly, for example: to check if customers are responding in the desired way.

What else can you learn from our conversation with Lukas Vermeer?

  • When to start experimenting and how to build a culture of experimentation
  • The importance of autonomy for experimentation teams
  • The three levels of experimentation: method, design, execution
  • How to accelerate the experimentation process
About Lukas Vermeer

Lukas Vermeer is an expert in implementing and scaling experimentation with a background in AI and machine learning. Currently, Lukas is the director of experimentation at Vista. Prior to this, he spent over eight years at Booking.com, from data scientist, product manager to director of experimentation. He continues to offer his expert consulting services to companies that are starting to implement experimentation. His most recently co-authored paper, “It Takes a Flywheel to Fly: Kickstarting and Keeping the A/B Testing Momentum,” helps companies get started and accelerate experimentation using the “investment follows value follows investment” flywheel.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Share on

LinkedIn Facebook Twitter

Article

Oct 20, 2021
11min read

Net Promoter Score (NPS): Your Ultimate Guide to the What, Why, and How

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

In a world where customers increasingly seek to buy into a brand than buy from a brand, it’s critical that companies create experiences that turn customers into loyal fans, rather than regard them as simple business transactions.

Customer satisfaction alone is no longer enough to thrive in today’s economy. The goal is to earn your customers’ fierce loyalty with authenticity and transparency, while aligning your offers and actions with a mission that speaks to them.

By measuring the net promoter score (NPS), businesses gain unique insight into how consumers perceive their customer journey in a number of different ways. Companies that use NPS to analyze customer feedback and identify areas of improvement hold the keys to optimizing rapid and effective business growth.

In this article, we’ll cover why measuring NPS is essential to scaling business sustainably, how to gather and calculate NPS feedback, and best practices to increase response rates and run successful NPS campaigns.

[toc]

What is NPS?

Let’s start with a little history. The Net Promoter Score was officially pioneered and coined by Fred Reichheld in the early 2000s, and has since become an invaluable methodology for traditional and online businesses alike. The value lies in using data to effectively quantify customer loyalty and its effect on business performance — a factor that was previously challenging to measure at scale.

What is NPS?
(Source)

The system works by asking customers a version of this question: How likely are you to recommend our brand/product/service to a friend or colleague? Answers range on a scale of 0-10, from “not at all likely” to “extremely likely.” Depending on their answers, respondents are separated into one of three categories.

  • Promoters (score 9-10): Loyal customers who keep buying and actively promote and refer your brand to their circle of friends, family, and/or colleagues.
  • Passives (score 7-8): Customers who’ve had satisfactory or standard experiences with your brand, and are susceptible to competitors’ offers.
  • Detractors (score 0-6): Unhappy customers who risk damaging your brand with public complaints and negative word-of-mouth.

To calculate the final net promoter score, subtract the percentage of promoters from the percentage of detractors. The metric can range from a low of -100 to a maximum of 100, the latter if every customer was a promoter.

For many e-commerce companies, high customer retention, referral, and positive reviews are all critical drivers of success. NPS helps these businesses understand overall buyer behaviors and identify which customer profiles have the potential to be brand enthusiasts, enabling marketers to adjust their strategy to convert passives into promoters.

Simply put, NPS surveys are a simple and powerful method for companies to calculate how customer experience management impacts their overall business performance and growth.

How to gather NPS feedback

Common methods used to gather NPS feedback are email, SMS, and website pop-ups or chat boxes. Regardless of which method is used, there is a common set of steps to ensure a successful NPS campaign:

  1. Set clear objectives before sending out the NPS survey. Save time and increase the relevance of survey responses by determining exactly what kind of feedback you’re looking for before launching the survey.
  2. Segment recipients with customer behavior profiles. Get specific with your survey questions by customizing them to different audiences based on their unique history and interaction(s) with your brand.
  3. Make surveys short, concise, and timely. Instead of lengthy annual or quarterly feedback requests, increase response rates by sending quick and easy surveys to customers soon after they’ve had meaningful interactions with your brand.
  4. Use an automation tool to optimize survey delivery. Whether it’s with an email marketing platform or website widget integration, using automation tools to design and deliver your NPS surveys streamlines the entire feedback process, while reducing the margin for human error.

Integrating the NPS survey directly into the customer journey on your website increases response rate and relevancy of feedback. To implement a NPS survey like this, try using an intuitive visual editor like AB Tasty with NPS widget capabilities.

AB Tasty’s visual editor enables marketers of all levels to:

  • Modify visual and interactive elements on the website without any manual coding necessary;
  • Set up action-tracking to directly measure the performance of variations you’ve created;
  • Use the NPS widget to customize the content and feel of surveys across one or more pages of the website; and
  • Track the evolution potential of customer loyalty and benchmark against competitor performance via the NPS report.

Below are two case studies of clients who’ve used the AB Tasty NPS widget with highly successful campaigns to collect customer feedback and gain valuable insight to improve their customer experiences.

How to calculate NPS feedback

So what makes a good NPS score? A general rule of thumb states that anything below 0 means your business has some work to do 
 and a “good score” falls between 0-30. However, the true value of a NPS score depends on several factors — namely what industry your business is in.

If your NPS score isn’t as high as you’d hoped, don’t fret! There is always room for improvement and the good news is that it’s easy to implement actionable changes to optimize your NPS campaigns, no matter where you are on the scale.

When benchmarking for NPS, look at competitors that are in the same industry and relatively similar size as your company to get the most accurate visualization possible. Look for graphs that map out average NPS data by industry to get more insights on performance and opportunities for improvement in your sector.

It’s important to understand that comparing your business’s results to significantly larger or unrelated brands can lead not only to inaccurate interpretation of the data, but also sets unrealistic and irrelevant goals for customer experience teams.

How to increase your NPS response rate

Reaching your customers with your NPS survey is just one half of the battle. The other half is getting enough customers to actually respond to it, which is critical to calculate an NPS score that accurately reflects your company’s customer satisfaction performance. Here are some tips for boosting your NPS response rate:

  • Customize your NPS survey. Take the time to brand your survey with the proper fonts and colors, following your brand design guide. Given the fact that the average person sees upwards of 6,500 ads in a day, information overload is a real struggle for consumers and marketers alike. A consistent look and feel from your survey helps customers recognize and trust your brand, making it an easy transition to take the next step in their customer journey.
  • Personalize the message. Studies show that personalized subject lines increase email open rates by 26%. If you’re sending the survey in an email, use merge fields or tags to automatically add each recipient’s name into the subject line or body of the email.
  • Use responsive design. 75% of customers complete surveys on their phone. Make sure your survey is fully functional and accessible from all devices (i.e., desktop, mobile, and tablet), as well as on as many operating systems and internet browsers as possible.
  • Offer incentives for completing the survey. From gift cards, cash, and promo codes to raffles, offering monetary rewards is an easy method to increase engagement, especially for longer surveys. However, this should be researched and done carefully to avoid review bias and more seriously, legal issues.

Why you should use NPS

Taking customer feedback seriously is important business. As of 2020, 87% of people read online reviews for local businesses, and 79% said they trust online reviews as much as a personal recommendation from friends or family. This means your customers’ perception of your brand can literally make or break it.

It’s clear that looking at sales revenue as the sole determiner of success is not sustainable for long-term business growth. Neither is assuming that several user case scenarios represent the majority without the data to prove it.

NPS is an especially powerful metric for e-commerce, as it uses data to help businesses identify truly relevant areas for improvement and opportunities to build a strong and loyal customer base that is so vital to thrive in this sector.

Build a strong relationship with your customer base
Building a strong relationship with your customer base and incentivizing brand promoters is crucial to succeeding in the e-commerce market

Rather than guesstimating what priorities should be, businesses can use longer surveys with open-ended questions to evaluate how their customers feel about specific aspects of the business (e.g., products, website, and brand) and target strategy accordingly.

When calculated correctly, NPS is the key to determining the likelihood of repeat business and acquisition driven by brand promoters. Marketing and product teams can boost customer retention and increase sales with customized products they know buyers want. Happy customers love loyalty programs and referral rewards, which also bring in new business with significantly less spend than cold advertising.

When is the ideal time to send users an NPS survey

Deciphering what time customers are most likely to open emails, or when they’re more responsive to brand communications, is one of the biggest challenges for marketing teams.

Some studies suggest that the best time of the week to send emails is Tuesday at 10am. Although as many marketers know from experience, a one-time-fits-all solution doesn’t truly exist (though we wish it did!).

Depending on your industry and audience, your brand’s ideal time to hit send will likely change over time — and experimentation and optimization are the best ways to stay on top of it.

Identify the right time to send customer satisfaction surveys
Identifying the right time to send customer satisfaction surveys requires continual testing of different elements like message personalization and audience segmentation

However it is possible to find ideal times based on data you likely already have: by focusing on meaningful interactions between brand and customer.

One of the optimal times to send a NPS survey is shortly after customers have had a meaningful interaction with the brand. This could be after a customer finishes a purchase cycle, receives a product, or even speaks with customer service.

During this time, the customer experience is still top-of-mind, which means they are more likely to complete a feedback survey with higher chances of providing more detailed — and honest — insights.

It’s also better to send short surveys more frequently. Asking for smaller amounts of feedback more often than once or twice a year enables you to monitor customer satisfaction with a quicker response time.

With regular feedback surveys, businesses can catch onto unhappy customers early on and make prompt changes to address problems in the customer journey, increasing customer retention.

Another benefit of this practice is that businesses can also identify highly successful campaigns throughout the year and prioritize resources on scaling strategies that are already proven to work well.

Do’s and don’ts for running an effective NPS campaign

Do:

  • Add open-ended questions. If you want more qualitative insight to support your business decisions, ask customers for specific input, as Eurosport did in this campaign.
  • Send from a person. Humans value real connections. Increase NPS response rate by sending surveys with the name and email of a real employee, not an automatic “no-reply” bot address.
  • Integrate your NPS survey into the user journey. To boost your reach beyond email surveys, use an NPS widget on your website for increased response rate and in-depth responses. Match your survey’s design to flow with the product page UX.

Don’t:

  • Disrupt the customer journey. Don’t overdo it with pop-up surveys or make them difficult to close, this can distract customers from their website experience and increase bounce rate.
  • Ask only one question. Don’t ask for just a 0-10 score. To collect actionable insight, add a follow-up question after the NPS score to ask why they gave that rating.
  • Not share NPS results. Transparency makes cross-team collaboration more effective and creative. NPS data is valuable for not only customer-facing teams, but also marketing and product teams to improve the customer experience.

Optimize your NPS strategy

In summary, NPS is incredibly user-friendly and simple to implement. This metric helps brands gain actionable insight into their customer loyalty and satisfaction, and identify opportunities to significantly boost customer retention and acquisition.

NPS widgets and automated feedback collection
NPS widgets and automated feedback collection enable cross-team collaborators to work more cohesively on customer experience campaigns

Businesses can use this data to run their operations better and smarter, and also improve cross-team collaboration on enhancing the customer experience. Regular testing and following best practices enable teams to continually improve their NPS strategy and reach higher response rates.

Ready to integrate your next NPS campaign directly into your website and customer journey? With an intuitive interface and no-code visual editor, AB Tasty enables you to fully customize the entire NPS survey live on your website, and experiment with different triggers to optimize your NPS strategy.

Our NPS widget makes it easy to scale this process quickly within even the fastest growing companies — give it a spin today.

AB Tasty’s NPS Widget Case Studies:

  1. How Eurosport’s Survey Pop-In Got 5K Responses in Less Than Two Weeks
  2. Avid Transforms Internal Culture and Website Experience with AB Tasty

Share on

LinkedIn Facebook Twitter

Article

Jul 28, 2021
3min read

Introducing 1,000 Experiments Club: A New Podcast Series From AB Tasty

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

Join VP Marketing Marylin Montoya as she takes a deep dive into all things experimentation

Today, we’re handing over the mic to AB Tasty’s VP Marketing Marylin Montoya to kick off our new podcast series, “1,000 Experiments Club.”

At AB Tasty, we’re a bunch of product designers, software engineers and marketers (aka Magic Makers), working to build a culture of experimentation. We wanted to move beyond the high-level rhetoric of experimentation and look into the nitty gritty building blocks that go into running experimentation programs and digital experiences.

Enter: “1,000 Experiments Club,” the podcast that examines how you can successfully do experimentation at scale. Our podcast brings together a selection of the best and brightest leaders to uncover their insights on how to experiment and how to fail 
 successfully.

In each episode, Marylin sits down to interview our guests from tech giants, hyper-growth startups and consulting agencies — each with their own unique view on how they’ve made experimentation the bedrock of their growth strategies.

You’ll learn about why failing is part of the process, how to turn metrics into your trustworthy allies, how to adapt experimentation to your company size, and how to get management buy-in if you’re just starting out. Our podcast is for CRO experts, product managers, software engineers; there’s something for everyone, no matter where you fall on the maturity model of experimentation!

We are kicking things off with three episodes, each guest documenting their journey of where they went wrong, but also the triumphs they’ve picked up from decades of experimentation, optimization and product development.

Ronny Kohavi (ex-Amazon, Airbnb, Microsoft) 

He shares a humbling reality check: Most ideas will fail 

Chad Sanderson (Convoy)

He breaks down the most successful types of experimentations 

Jonny Longden (Journey Further)

He believes anyone can and should do experimentation

In the culture of experimentation, there’s no such thing as a “failed” experiment: Every test is an opportunity to learn and build toward newer and better ideas. So have a listen and subscribe to “1,000 Experiments Club” on Apple Podcasts, Spotify or wherever you get your podcasts.

Take to me to the podcast!

Share on

LinkedIn Facebook Twitter

Article

Jul 19, 2021
5min read

1,000 Experiments Club: A Conversation With Jonny Longden of Journey Further

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

Is experimentation for everyone? A resounding yes, says Jonny Longden. All you need are two ingredients: A strong desire and tenacity to implement it.

There’s a dangerous myth lurking around, and it’s the idea that you have to be a large organization to practice experimentation. But it’s actually the smaller companies and start-ups that need experimentation the most, says Jonny Longden of performance marketing agency Journey Further.

With over a decade of experience in conversion optimization and personalization, Jonny co-founded Journey Further to help clients embed experimentation into the heart of what they do. He currently leads the conversion division of the agency, which also focuses on PPC, SEO, PR — among other marketing specializations.

Any company that wants to unearth any sort of discovery should be using experimentation, especially start-ups who are in the explorative phase of their development. “Experimentation requires no size: It’s all about how you approach it,” Jonny shared with AB Tasty’s VP Marketing Marylin Montoya.

Here are a few of our favorite takeaways from our wide-ranging chat with Jonny.

.

The democratization of experimentation

People tend to see more experimentation teams and programs built at large-scale companies, but that doesn’t necessarily mean other companies of different sizes can’t dip their toes in the experimentation pool. Smaller companies and start-ups can equally benefit from this as long as they have the tenacity and capabilities to implement it.

You need to truly believe that without experimentation, your ideas won’t work, says Jonny. There are things that you think are going to work and yet they don’t. Conversely, there are many things that don’t seem like they work but actually end up having a positive impact. The only way to arrive at this conclusion is through experimentation.

Ultimately, the greatest discoveries (for example, space, travel, medicine, etc.) have come from a scientific methodology, which is just observation, hypothesis, testing and refinement. Approach experimentation with this mindset, and it’s anyone’s game.

Building the right roadmaps with product teams

Embedding experimentation into the front of the product development process is important, but yet most people don’t do it, says Jonny. From a pure business perspective, it’s about trying to de-risk development and prove the value of a change or feature before investing any more time, money and bandwidth. 

Luckily, the agile methodology employed by many modern teams is similar to experimentation. Both rely on iterative customer collaboration and a cycle of rigorous research, quantitative and qualitative data collection, validation and iteration. The sweet spot is the collection of both quantitative and qualitative data — a good balance of feedback and volume. 

The success of building a roadmap for an experimentation program comes down to understanding the organizational structure of a company or industry. In SaaS companies, experimentation is embedded into the product teams; for e-commerce businesses, experimentation fits better into the marketing side. Once you’ve determined the owner and objectives of the experimentation, you’ll need to understand whether you can effectively roll out the testing and have the right processes in place to implement results of a test.

Experimentation is, ultimately, innovation

The more you experiment, the more you drive value. Experimentation at scale enables people to learn and build more tests based on these learnings. Don’t use testing to only identify winners because there’s much more knowledge to be gained from the failed tests. For example, you may only have 1 in 10 tests that work. The real value comes in the 9 lessons you’ve acquired, not just the 1 test that showed positive impact. 

When you look at it through these lenses, you’ll realize that the post-test research and subsequent actions are vital: That’s where you’ll start to make more gains toward bigger innovation. 

Jonny calls this the snowball effect of experimentation. Experimentation is innovation — when done right. At the root, it’s about exploring and seeing how your customers respond. And as long as you’re learning from the results of your tests, you’ll be able to innovate faster precisely because you are building upon these lessons. That’s how you drive innovation that actually works.

What else can you learn from our conversation with Jonny Longden?

  • Moving from experimentation to validation
  • How to maintain creativity during experimentation
  • Using CRO to identify the right issues to tackle
  • The required building blocks to successful experimentation
About Jonny Longden

Jonny Longden leads the conversion division of Journey Further, a performance marketing agency specializing in PPC, SEO, PR, etc. Based in the United Kingdom, the part-agency, part-consultancy helps businesses become data-driven and build experimentation into their programs. Prior to that, Jonny dedicated over a decade in conversion optimization, experimentation and personalization, working with Sky, Visa, Nike, O2, Mvideo, Principal Hotels and Nokia.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Share on

LinkedIn Facebook Twitter

Article

10min read

How to Release Fearlessly

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

The excitement of launching a new feature

Are you on the brink of launching a new feature – one that will affect many of your high-value clients? You’ve worked hard to build it, you’re proud of it and you should be! 

You can’t wait to release it for all your users, but wait! What if you’ve missed something? Something that would ruin all your engineering efforts? 

There’s nothing worse than starting the day after a release by having to immediately deal with a number of alerts for production issues and spending the day checking a number of logging and monitoring systems for errors and, ultimately, having to rollback the feature you just launched. You would just feel frustrated and unmotivated.

In addition to sapping the morale of your technical teams, NIST has shown that the longer a bug takes to be detected, the more costly it is to fix. This is illustrated by the following graph: 

Relative cost to fix bugs.

This is explained by the fact that once the feature has been released and is in production, finding bugs is difficult and risky. In addition to preventing users from being affected by problems, it’s critical to ensure service availability.

Are you sure your feature is bug-free?

You might think that this won’t happen to you. That your feature is safe and ready to deploy. 

History has shown that it can happen to the biggest companies. Let’s name a few examples.  

Facebook, May 7, 2020. An update to Facebook’s SDK rolled out to all users, missed a bug: a server value that was supposed to provide a dictionary of things was changed to provide a simple YES/NO instead. This really tiny change was enough to break Facebook’s authentication system and affect tons of other apps like TikTok, Spotify, Pinterest, Venmo, and other apps that didn’t even use Facebook’s authentication system as it is extremely common for apps to connect to Facebook regardless of whether they use a Facebook-related feature, mainly for ad attribution. The result was unequivocal, the app simply crashed right after launch. Facebook fixed the problem in a hurry, with about two hours for things to get back to normal. But do you have the same resources as Facebook?

Apple, September 19, 2012. Another good example, even though it’s a bit older, would be the replacement of Google Maps with Apple Maps in iOS 6 in 2012 on iOS devices. For many customers and especially fans, Apple always handles the rollout of new features carefully, but this time they messed up. Apple didn’t want to be tied to Google’s app anymore, so they made their own version. However, in their rush to release their map system, Apple made some unforgivable navigational mistakes. Among the many failures of Apple Maps are erased cities, disappearing buildings, flattened landmarks, duplicate islands, distorted graphics, and erroneous location data. A large part of this mess could have been avoided if they had deployed their new map application progressively. They would have been able to spot the bugs and quickly fix them before massive deployment.

And now, thinking about this and seeing that even big companies are impacted, you’re stressed out and may not even want to release it anymore.

But don’t worry! At AB Tasty, we know that building a feature is only half of the story and that to be truly effective, that feature has to be well deployed.

Our feature management service has you covered. You’ll find a set of useful features, such as progressive rollout, to free you from the fear of a release catastrophe and erase feature management frictions, so that you can focus on value-added tasks to get high-quality features into production and apply your energy and innovation in the best way possible, thereby delivering maximum value to your customers.

What’s progressive rollout? 

So now you’re curious: what’s progressive rollout? How will this help me monitor the release and make sure everything is okay?

A progressive rollout approach lets you test the waters of a new version with a restricted set of clients. You can set percentages of users to whom your feature will be released and gradually update the percentage to safely deploy your feature. You can also do a canary launch by manually targeting several groups of people at various stages of your rollout.

This is a practice already used by large companies that have realized the significant benefits of a progressive rollout. 

Netflix, for example, is one of the most dynamic companies and its developers are constantly releasing updates and new software, but users rarely experience downtime and encounter very few bugs or issues. The company is able to deliver such a smooth experience thanks to sophisticated deployment strategies, such as Canary deployment and progressive deployment, multiple staging environments, blue/green deployments, traffic splitting, and easy rollbacks to help development teams release software changes with confidence that nothing will break. 

Disney is another good example of a company that makes the most of progressive deployment. It has taken the phased deployment approach to a whole new level for its “Disney +” and “Star” streaming services by deploying them regionally rather than globally. This delivery method is driven by the needs of the business. The company is making sure that everything is ready at the regional level, in line with its focus on the most important markets. Prior to launching Disney+ in Europe, it spent a lot of time building the local infrastructure needed to deliver a high-quality experience to consumers when launching Disney+ in Europe, including establishing local colocation facilities and beefing up data centers to cache content regionally. After starting to roll it out in Europe, Disney was able to identify that, for some markets, the launch of Disney+ could actually create issues that would have resulted in latency and thus provide a poor experience for affected users. So they took proactive steps to reduce their overall bandwidth usage by at least 25% prior to their march 24 launch and delayed their launch in France by two weeks. Without progressive deployment, they wouldn’t have been able to identify these issues. And that’s why the launching of Disney + was remarkable.

What are the benefits of the progressive rollout?

There are three main benefits to the progressive rollout approach.

Avoiding bugs impacting everyone in production at once

First, by slowly ramping up the load, you can monitor and capture metrics about how the new feature impacts the production environment. If any unanticipated issues come to light, you can pause the full launch, fix the issues, and then smoothly move ahead. This data-driven approach guarantees a release with maximum safety and measurable KPIs.

Validating the “Viable part” in your MVP

You can effectively measure how well your feature is welcomed by your users. If you launch a new feature to 10% of your client base and notice revenue or engagement taking a dip, you can pause the release and investigate. The other major advantage? Anticipating costs. Since margin, profit and revenue are an important part of sustainability, unexpected costs that blow up your projected budgets at the end of the month are almost as bad as the night sweats that come from an unexpected bug! Monitoring your costs during a progressive rollout and immediately pausing the launch if those costs spike is a phenomenal level of control that you will absolutely want to get in on.

Progressively deploying services based upon business drivers

Finally, deploying a service or product progressively can also be seen as a way of prioritizing specific markets based on data-driven business plans. Disney, for example, decided not to launch the service in the U.S. when it launched “Star,” its new channel available in the Disney+ catalog for international audiences, which will feature more mature R-rated movies, FX TV shows, and other shows and movies that Disney owns the rights to but that do not fit the Disney+ family image. Ironically: U.S. customers will have to pay extra on their Disney+ subscription to access the same content on the other streaming service, Hulu.

The decision was made following a complex matrix of rights agreements and revenue streams. Disney found that subscribers are willing to pay for the separate Hulu and Disney+ libraries in the U.S., but that Star’s more limited lineup was enough to justify a standalone paid purchase for international customers, who will have to add $2 to their initial $6.99 subscription to access it. When the content library for Star is enough to justify not going through Hulu anymore, the U.S. customers will have access to it by paying just 1$ more. This progressive rollout approach has enabled Disney to make sure that once they launch Star in the U.S., everything will be ready and they will achieve good results.

In other words, the progressive rollout approach helps you ensure that your functionality meets the criteria of usability, viability, and desirability in accordance with your business plan.

How to act fast when you identify bugs while progressively deploying a feature?

Now that you know more about the progressive rollout of your features/products, you may be wondering how to take action if you identify bugs or if things aren’t going well. Lucky for you, we’ve thought of that part too. In addition to progressive rollout, you’ll also find automatic rollback on KPIs and feature flagging in the AB Tasty toolkit.

Feature flagging will let you set up flags on your feature, that work as simply as a switch on/off button. If for any reason you identify threats in your rollout or if the engagement of your users is not really convincing, you can simply toggle your feature off and take time to fix any issues. 

This implies that you are aware and that someone from the product team is available to turn it off. But what if something happens overnight and no one can check on the progress of the deployment? Well, for that eventuality, you can set up automatic rollbacks (also called Rollback Threshold) linked to key performance indicators. Our algorithm will check the performance of your deployment and, based on the KPIs you set, if something goes wrong, it will automatically roll back the deployment and inform you that a problem has occurred. This way, in the morning, your engineers will be able to fix the problems without having to deal with the rollback themselves.

Conclusion

Downtime incidents are stressful for both you and your customers. To resolve them quickly and efficiently, you need to have access to the right tools and make the most of them. The progressive rollout, automatic rollback, and feature flagging are great levers to relieve your product teams of stress and let them focus on innovating your product to create a wonderful experience for your users. Highly effective organizations have already realized the importance of having the right approach to deployment with the right tools. What about your organization? 

AB Tasty minimizes risk and maximizes results to make the lives of Product teams a whole lot easier. Create a free account today!

Share on

LinkedIn Facebook Twitter

Article

Jul 2, 2021
5min read

1,000 Experiments Club: A Conversation With Chad Sanderson of Convoy

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

Chad Sanderson breaks down the most successful types of experimentations based on company size and growth ambitions

For Chad Sanderson, head of product – data platform at Convoy, the role of data and experimentation are inextricably intertwined.

At Convoy, he oversees the end-to-end data platform team — which includes data engineering, machine learning, experimentation, data pipeline — among a multitude of other teams who are all in service of helping thousands of carriers ship freight more efficiently. The role has given him a broad overview of the process, from ideation, construction to execution.

As a result, Chad has had a front-row seat that most practitioners never do: The end-to-end process of experimentation from hypothesis, data definitions, analysis, reporting to year-end financials. Naturally, he had a few thoughts to share with AB Tasty’s VP Marketing Marylin Montoya in their conversation on the experimentation discipline and the complexities of identifying trustworthy metrics.

Introducing experimentation as a discipline

Experimentation, despite all of its accolades, is still relatively new. You’ll be hard pressed to find great collections of literature or an academic approach (although Ronny Kohavi has penned some thoughts on the subject matter). Furthermore, experimentation has not been considered a data science discipline, especially when compared to areas of machine learning or data warehousing.

While there are a few tips here and there available from blogs, you end up missing out on the deep technical knowledge and best practices of setting up a platform, building a metrics library and selecting the right metrics in a systematic way.

Chad attributes experimentation’s accessibility as a double-edged sword. A lot of companies have yet to apply the same rigor that they do to other data science-related fields because it’s easy to start from a marketing standpoint. But as the business grows, so does the maturity and the complexity of experimentation. That’s when the literature on platform creation and scaling is scant, leading to the field being undervalued and hard to recruit the right profiles.

When small-scale experimentation is your best bet

When you’re a massive-scale company — such as Microsoft or Google with different business units, data sources, technologies and operations — rolling out new features or changes is an incredibly risky endeavour, considering that fact that any mistake could impact millions of users. Imagine accidentally introducing a bug for Microsoft Word or PowerPoint: The impact on the bottom line would be detrimental.

The best way for these companies to experiment is with a cautious, small-scale approach. The aim is to focus on immediate action, catching things quickly in real time and rolling them back.

On the other hand, if you’re a startup in a hyper-growth stage, your approach will vastly differ. These smaller businesses typically have to show double-digit gains with every new feature rollout to their investors, meaning their actions are more so focused on proving the feature’s positive impact and the longevity of its success.

Make metrics your trustworthy allies

Every business will have very different metrics depending on what they’re looking for; it’s essential to define what you want before going down the path of experimentation and building your program.

One question you’ll need to ask yourself is: What do my decision-makers care about? What is leadership looking to achieve? This is the key to defining the right set of metrics that actually moves your business in the right direction. Chad recommends doing this by distinguishing your front-end and back-end metrics: the former is readily available, the latter not so much. Client-side metrics, what he refers to as front-end metrics, measure revenue per transaction. All metrics then lead to revenue, which in and of itself is not necessarily a bad thing, but that just means all your decisions are based on revenue growth and less on proving the scalability or winning impact of a feature.

Chad’s advice is to start with the measurement problems that you have, and from there, build out your experimentation culture, build out the system and lastly choose a platform.

What else can you learn from our conversation with Chad Sanderson?

  • Different experimentation needs for engineering and marketing
  • Building a culture of experimentation from top-down
  • The downside of scaling MVPs
  • Why marketers are flagbearers of experimentation
About Chad Sanderson

Chad Sanderson is an expert on digital experimentation and analysis at scale. He is a product manager, writer and public speaker, who has given lectures on topics such as advanced experimentation analysis, the statistics of digital experimentation, small-scale experimentation for small businesses and more. He previously worked as senior program manager for Microsoft’s AI platform. Prior to that, Chad worked for Subway’s experimentation team as a personalization manager.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Share on

LinkedIn Facebook Twitter

Article

Jun 30, 2021
5min read

1,000 Experiments Club: A Conversation With Ronny Kohavi (Formerly of Airbnb, Microsoft and Amazon)

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

One of the pioneers of experimentation shares a humbling reality check: Most ideas will fail (and it’s a good thing)

Few people have accumulated as much experience as Ronny Kohavi when it comes to experimentation. His work at tech giants such as Amazon, Microsoft and Airbnb — just to name a few — has laid the foundation of modern online experimentation. 

Before the idea of “build fast, deploy often” took hold across tech companies, developers followed a waterfall model that saw fewer releases (sometimes every 2-3 years). The shortening of development cycles in the early 2000s thanks to the Agile methodology and an uptick in online experimentation created the perfect storm for a software development revolution ― and Ronny was at the center of it all. 

AB Tasty’s VP Marketing Marylin Montoya set out to uncover the early days of experimentation with Ronny and why failure is actually a good thing. Here are some of the key takeaways from their conversation.

.

Progressive deployments as a safety net 

A typical cycle of experimentation involves exposing the test to 50% of the population for an average of two weeks before a gradual release. But Ronny suggests coming at it from a different vantage point: Starting with a small audience of just 2% before ramping up to 50%. The slower ramp-up gives you the time to detect any egregious issues or a degradation in metric values in near real time. 

In an experiment, we may focus on just two features, but we have a large set of guardrails that suggest we shouldn’t be degrading X, Y or Z. Statistical data that you’re collecting could also suggest that you’re impacting something you didn’t mean to. Hence, the usage of progressive deployments in which you can identify external factors and easily rollback your test.

It’s like if you’re cooling water: You may realize you’re changing the temperature, but it’s not until you reach 0ÂșC (32ÂșF) that ice forms. You suddenly realize that when you get to a certain point, something very big happens. So, deploying at a safe velocity and monitoring the results can lead to huge improvements.

Your great idea? It will most likely fail.

Nothing gives you a better reality check than experimentation at scale. Everyone thinks they’re doing the best stuff in the world until it’s in the hands of their users. That’s when the real feedback kicks in.

Over two-thirds of ideas actually fail to move the metrics that they were designed to improve — a statistic Ronny shares from his time at Microsoft, where he founded the experimentation platform team of more than 100 data scientists, developers and program managers. 

Don’t be deterred, however. In the world of experimentation, failing is a good thing. Fail fast, pivot fast. Being able to realize that the direction you’re going in isn’t as promising as previously thought enables you to use those new findings to enrich your next actions.

At Airbnb, Ronny’s experimentation team deployed a lot of machine learning algorithms to improve search. Out of 250 ideas tested in controlled experiments, only 20 of them proved to have a positive impact on the key metrics — meaning over 90% of ideas failed to move the needle. On the flip side, however, the 20 ideas that did succeed in some form? Those resulted in a 6% improvement in booking conversion, worth hundreds of millions of dollars.

The starter kit to experimentation

It’s easier today to convince leadership to invest in experimentation because there are plenty of successful use cases out there. Ronny’s advice is to start with a team that has iteration capital. If you’re able to run more experiments and a certain percentage are pass/fail, this ability to try ideas is key. 

Pick a scenario where you can easily integrate the experimentation process into the development cycle and then work your way on to more complex scenarios. The value of experimentation is clearer because deployments are happening more often. If you’re working in a team that deploys every six months, there’s not a lot of wiggle room because everyone has already invested their efforts into this idea that the feature cannot fail. Which, as Ronny pointed out earlier, has a low probability of success.

Is experimentation for every company? The short answer is no. A company has to have certain ingredients in order to unlock the value of experimentation. One ingredient you need is being in a domain where it’s easy to make changes, such as website services or software. A second ingredient is you need enough users. Once you have tens of thousands of users, you can start experimenting and doing it at scale. And lastly, make sure you have trustworthy results from which you are taking your decisions.

What else can you learn from our conversation with Ronny Kohavi?

  • How experimentation becomes central to your product build 
  • Why experimentation is at the root of top tech companies 
  • The role leaders play in evangelizing an experimentation culture
  • How to build an environment for true experimentation and trustworthy results
About Ronny Kohavi

Ronny Kohavi is an authority in experimentation, having worked on controlled experiments, machine learning, search, personalization and AI for nearly three decades. Ronny previously was vice president and technical fellow at Airbnb. Prior to that, Ronny led the Analysis and Experimentation at Microsoft’s Cloud and AI group and was the director of data mining and personalization at Amazon. Ronny has also co-authored “Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing.,” which is currently the #1 best-selling data-mining book on Amazon.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Share on

LinkedIn Facebook Twitter