A suggested best practice in A/B testing. A/A testing is used as a dry run before a true A/B test, to ensure that the testing tool is functioning correctly (well configured). If the test has been accurately set up, the A/A test should show no significant difference in performance between the two versions. A/A tests may also be useful to determine a baseline KPI with which to compare A/B test results, or to have an idea of the amount of traffic needed to establish a statistically reliable A/B test. A/A testing is generally recommended for websites with high traffic volume.
A simple CRO method that consists of showing a website visitor one of two versions of a website page (or email, ad copy, etc.,) version A (usually the control) or version B. Half of the target audience is randomly assigned to version A, and the other half to version B. Only one element in version B is different from that of version A, allowing those who run the test to attribute changes in behavior to this one attribute. For example, one might change only the wording of a CTA, or the color of a button, or the hero image on a page, and so on. Differences in behavior of the target audience towards the two versions are established according to statistically reliable tests, and tracked in order to determine which performs better according to a predefined set of KPIs.
Become an expert:
The A/B Testing Complete Guide
Above the Fold
Originally used in the context of print media, in a digital context, ‘above the fold’ denotes the area on a webpage that is immediately visible to the viewer, without the viewer having to scroll down. Screen size and device (iPad, smartphone…) will affect what content is considered ‘above the fold.’ Best practice dictates that the most crucial, eyecatching or interesting visual content should be placed above the fold, to increase viewer engagement and lower bounce rates.
Average Order Value
Average Order Value (or AOV) refers to the average total of all orders placed via an ecommerce site over a given period of time. Increasing AOV is often the goal of a CRO strategy. Average Order Value can be calculated by dividing revenue by number of orders.
Calculate and Increase your Average Order Value
A statistical approach based on the work of Thomas Bayes (1702-61). When referring to A/B testing: As opposed to the Chi² and other "frequentist" statistical methods, (t-test, z-test, etc...), which only qualify the size of the difference between A and B, a Bayesian statistical approach provides a confidence interval around the size of the difference between variations. Essentially, a Bayesian statistical approach incorporated into an A/B testing tool allows the tester not only to know a purported difference in variation between version A and B, but also the statistical likelihood of these differences in performance.
A technique used in digital ad marketing, where ads are served to browsers according to their online behavior, such as number and type of pages visited and search history. A ‘profile’ of an online browser is built up and used to show them more relevant, personalized ads with the aim of bringing the along the conversion funnel.
Below the Fold
‘Below the fold’ denotes the area on a webpage that is not immediately visible to the viewer, i.e. the area of the page a viewer must scroll down to reach. Content placed below the fold is considered to have a lower chance of being viewed and engaged with, since it is less readily accessible.
In web marketing, when an internet user arrives on the page and then exits the website without visiting any other page of the website. In email marketing, may also refer to when an email is unable to be delivered (see soft and hard bounce). Used to calculate bounce rate, or the percentage of visitors who arrive on a site and then leave without viewing any other pages on the site. Often a KPI used to indicate interest in or effectiveness of a page.
How to Improve your Bounce Rate
A fictional, generalized representation of a company’s target audience. Buyer personas are used to guide content, product and service packaging creation, by clearly outlining a target audience’s needs, particularities and defining characteristics.
Use Buyer Persona to Personalize your Website
An attribute of a marketing campaign - usually a button or a link - that incites the viewer to complete a desired action (e.g. buy now or click here). CTAs are important elements of a marketing campaign, since they are directly responsible for conversions.
How to make people click on your CTA
Click-Through Rate (CTR)
Usually, the number of times an ad is clicked on per ad impression, expressed as a percentage. May also refer more generally to any element that is clicked on per impression, for example, how many times a link is clicked on in an email, divided by the number of times the email was opened.
How to Improve Click Through Rate
The systematic tendency, used by the brain as an information processing shortcut, to base judgment, memory, decision-making, etc., on one’s personal frame of reference instead of on rational logic. In a UX context, it is important to take cognitive biases into account to anticipate how, for example, website visitors might react to a certain page design. Examples of cognitive bias include the recency effect, or the tendency for people to remember a piece of information that is last in a sequence, the bandwagon effect, which dictates that people tend to believe something if it is demonstrated that many other people believe the same thing, and confirmation bias, the tendency to believe or seek information that confirms existing beliefs. Many cognitive biases can also be used by CRO experts to increase conversions.
Also, confidence index. The confidence interval is a way of indicating the level of uncertainty inherent in the CTR results of any A/B testing tool. Confidence intervals are used to qualify CTRs as well as the lift of observed CTRs between variations.
This a general marketing term that refers to when a desired action is completed by the subject of a marketing campaign. Common conversions include when someone subscribes to a newsletter, downloads a piece of content, clicks on an ad, etc.
Also, sales funnel. The path a website visitor takes through a website which leads to a desired final conversion, usually a macro-conversion. Conversion funnels are often referred to in the context of ecommerce websites, but may be applied to any online context in which prospects move through a process towards a final desired action.
Conversion Rate Optimization (CRO)
The discipline of optimizing a marketing campaign or UX so that it will yield the maximum amount of desired conversions. CRO involves getting to know your target audience and the frequent processes they encounter that surround the completion of a desired action, i.e. site navigation, scrolling through an email or social network feed, etc. A variety of methods exist to aid marketers with CRO.
Customer Lifetime Value (CLV)
A metric used to calculate the profit any given customer brings to a business. Customer Lifetime Value can help you answer questions like, how much the marketing department should spend to acquire a customer, how much customer support should spend to satisfy and retain customers, and, what kind of prospects are the most interesting for sales teams.
How to Calculate Customer Lifetime Value
Not to be confused with personalization. First identified in 1987 in the book, Future Perfect by Stan Davis, ‘mass customization’ has come to mean the process of allowing customers to assemble goods and services that they modify to fit their specific tastes or needs, at scale. An example might be a fashion retailer that allows customers to choose from a range of color, fabric, and style patterns to piece together a customized shoe via the retailer’s website.
Customer Journey Map
A diagram, chart or illustration that depicts the different, sequential stages customers go through when engaging with a company. They are often used to try and make the conversion funnel smoother and increase conversion rates, by improving the customer experience.
Data Management Platform (DMP)
A Data Management Platform is used to combine and manage data from different sources. Offline and online data from web applications, web analysis tools, CRM systems or even in-store purchases can be collected, processed and actively used by different third party tools like A/B testing or personalization platforms. In general, cookie IDs are collected from various systems, which are then used to create user profiles and ultimately to target groups using segmentation.
A marketing tactic used to re-engage website visitors after shopping cart abandonment. Transactional emails are sent to website visitors who added items to their cart but did not finalize the purchase. A typical email remarketing workflow might include an initial reactivation email reminding the shopper of the contents of their cart, and a second email offering an incentive like a promotion or discount code to encourage the finalization of the purchase.
Email A/B Testing consists of applying principles of A/B testing two versions of a web page, to two versions of an email campaign. One might change the headline, content, layout, CTA, or any other number of elements to decide which performs best in terms of KPIs like open or click rate.
A Beginner’s Guide to A/B Testing your Emails
Sometimes confused with bounce rate. Exit rate is a digital marketing term used to indicated the percentage of visitors to a site who actively click away to a different site from a specific page. These visitors may have visited other pages on the site prior to exiting, which distinguishes them from ‘bounced’ visitors.
Also known as FOOC (Flash of Original Content). In a testing context, the undesirable effect that can occur when the original version of a page appears briefly before the modified page is shown - causing the website visitor to experience a ‘flickering’ of the page. Flickering is caused when the website browser cannot process the changes made to a page fast enough. Although there is no surefire way to avoid flickering, there are some commonly accepted best practices to help avoid it as much as possible, including optimizing your site’s loading time, placing an A/B testing solution tag as high as possible in the source code, and avoiding using a tag manager.
In email marketing, when an email is undeliverable due to a permanent reason, which might include the fact that the recipient’s email address does not exist, the domain name does not exist, or the recipient’s email server has completely blocked delivery.
A form of testing (website optimization) that consists of testing certain online media headlines against each other for various audience segments, to determine which stimulates the most engagement.
Heat maps, like the one to the right, are visual representations of attention, engagement and interactions generated by visitors as they navigate through a website. Types of heat maps include click heat maps, (which represent activity in warm colors, and lack of activity in cool colors), scroll-maps (which show how far down website visitors scroll), ‘percentage of clicks’ heat maps (which illustrates, element by element, how many clicks were generated), and confetti heat maps (which let you see each individual click on a page, as opposed to a view that shows a ‘density’ of clicks). Heat maps are often used to find useful insights for CRO.
How Heatmap Can Help You Improve Your Website
Key Performance Indicator (KPI)
Key Performance Indicators, are quantifiable metrics, such as pageviews, clicks or downloads, that are used to assess the level of performance of a given marketing campaign. For example, a KPI for a newsletter emailing might include open rate and click-through rate.
How to Define and Track Your KPIs
Usually, the page a website visitor is directed to after clicking on an ad. A landing page is a website page specially designed to target a particular type of visitor according to specific demographic, interests and buying behavior criteria. It aims to attract leads as effectively as possible (in B2B and B2C modes) and to maximize the conversion of visitors into buyers.
Become an expert:
Landing Pages: Tips & Best Practices
Also, algorithm or AI-based personalization. A type of personalization in which marketers set up algorithms and machine learning technologies which themselves dynamically present website visitors with what the algorithms consider to be the most appropriate message. Machine-learning personalization is appropriate for large scale operations, and marketers don’t necessarily stay in control of which audiences are exposed to what message.
The overall goal of your website, and the final desired action you would like your website visitors to make. Macro-conversions are often lead up to by microconversions. Macro-conversion examples might include payment confirmation for an online cart for an ecommerce retailer, subscribing to an online service for a digital media site, or filling out a form for a lead generation site.
A micro-conversion is an action taken by a website visitor that, though itself not a revenue-generating conversion, is either part of the conversion funnel to a final, overall conversion goal (macro-conversion), or is somehow related to a final conversion goal. Examples of micro-conversions might include viewing the pricing page, adding an item to the cart, requesting a quote, downloading a client case study, commenting on a company blog article, etc.
A form of A/B testing in which machine learning algorithms attribute traffic to a winning variation earlier than would be possible with traditional A/B testing, in order to save time and maximize positive outcomes. The name comes from the ‘gamble’ of using slot machines (‘one-armed bandits’) to determine which will earn you the most.
As opposed to A/B testing, multivariate testing is a CRO method in which one can test a combination of different attributes on a target audience to determine which combination performs the best, according to predefined KPIs. Common examples include changing the color, shape and wording of a CTA to determine which has the highest CTR.
Become an expert:
All You Need to Know About Multivariate Testing
Online marketing actions that aim to reduce shopping cart abandonment or encourage a website visitor to continue browsing. Common on-site marketing tactics include showing a retention pop-in when a visitor is about to leave a cart or payment page without confirming their order, that encourages the visitor to complete their order or leave their email address to stay up to date with items they’re interested in. Marketers often use incentives in their on-site remarketing actions, like offering free shipping over a certain order amount, or offering a promo code, to encourage payment confirmation.
The process, usually in a marketing context, of making prospects’ or clients’ experiences more relevant, based on their characteristics (behavioral, demographic…). Examples include segmenting email marketing campaigns based on the recipient’s industry, job title or place in the sales funnel, personalizing an e-commerce website so that women in Spain see a promotion on bikinis, and men in Canada see discounts on men’s outerwear (see ‘website personalization’), product recommendations based on previous search patterns, etc. The goal is to increase engagement - and therefore conversion rates, whether measured in CTR, sign ups, Average Order Value or yearly revenues.
A marketing technique that involves using algorithms to predict, based on previous behavior, the items a website visitor might be interested in, and showing these to the browser in the hopes that they will convert.
The anxiety associated with making a purchase. Common purchase anxieties include worry about the efficacy of the product, trustworthiness of the vendor with bank details and case, concern over shipping reliability and return policy. A variety of CRO technique can be used to decrease purchase anxiety.
The Science Behind Purchase Anxiety
Revenue per Visitor
This KPI reflects the average revenue per visitor to your website, and is used to measure how well online sales are performing. RPV is calculated by dividing total income by number of visitors during a specific time period. For example, if your income for January to March is $20,000, during which time you attracted 5,000 visitors, then your RPV would be $4.
How to Measure Revenue per Visitor
A type of personalization, rule-based personalization relies on the manual creation and management of business rules for deploying personalized messages to specific audience segments. The process goes, If someone from segment X triggers Y action, show them message Z. Marketers stay in control of what segments see what messages with rulebased personalization.
An ecommerce neologism created by combining ‘search’ and ‘merchandizing’. The marketing tactic of using an online search function to promote products related to the keyword or phrases entered in the search bar by the user. Related to faceted navigation, faceted browsing, multi-faceted search.
A tactic used in personalization. Segmentation is the process of grouping together individuals according to common, meaningful characteristics, such as geographic location, gender, buying behavior, etc, in order to deliver more relevant marketing messages.
How to Segment Your Audience
Server-Side A/B Testing
Server-side A/B testing directly uses your server to perform the test, sending the modified page or element to the target audience. Though it requires direct involvement of an IT team, (and so isn’t an optimal choice for marketers new to testing), server-side A/B testing opens up more avenues for experimentation, thanks to its direct connection with your back office. Testing on different devices (mobile apps, IoT) and in more complex ways is often only possible server-side.
Like heat mapping, session recording is a web marketing tactic used to gather insights about how website users engage with a site. It involves anonymously recording the mouse movements of a website visitor in order understand their interaction with the page. Often used in the context of CRO.
User Session Replay and Session Recording Tool
Shopping Cart Abandonment
When users who’ve added a product(s) to their shopping cart, but proceed to leave the site before finalizing their purchase. Often a KPI measured in CRO strategies. Common reasons for shopping cart abandonment include higher than expected shipping costs, order value not being high enough to qualify for free shipping, lack of available preferred payment methods, resistance to creating an account with the merchant, and the idea that many users add products to the cart merely to get an idea of total cost or to use as a shopping list, without the intent to buy in the moment.
Social proof, also called the Bandwagon Effect, is the tendency to do something because other people are doing it. In the context of CRO, showing good product reviews or ‘low in stock’ messaging is a way of saying, ‘many people are buying this, and so should you.’
In email marketing, when an email is undeliverable due to a temporary reason. This may be because the recipient's mailbox is full, their email server is down or offline, or the email is too large.
Become an expert:
A Guide to Becoming a Split Testing Expert
In the context of A/B, multivariate or other kinds of testing, the point at which the confidence index is equal to or greater than a given threshold. Theory dictates that this threshold is fixed once, before the start of the experiment. A conventional threshold for statistical significance is 95%, meaning that, statistically, 1 in every 20 results will be wrong. This threshold should be set with the distinctive characteristics of each business in mind, as it is directly linked to the risk deemed reasonable for the experiment.
The idea or hypothesis being tested during a website optimization test. For example, one test hypothesis might be that, by changing the color of a CTA from black to yellow, there will be more engagement with the CTA, since it will be more visible.
How to Formulate a Strong A/B Test Hypothesis?
A marketing tactic which relies on creating a sense of urgency in order to encourage sales or engagement. ‘Only X number of items left’ messaging on websites, or ‘X number of people are also looking at this hotel rooms,’ are common examples of urgency marketing.
User Experience (UX)
A general term used to describe how easy or agreeable it is for someone to use or interact with a given product, website, application, company or similar. An essential aspect of CRO.
Usually, the reaction of users of a digital property, such as a website or mobile app, on it’s usability and user interface. User feedback is integral to building a solid product roadmap and also to finding conversion rate optimization test ideas.
Usability tests are processes designed to observe and track real users while they use a product to measure its usability and user-friendliness in order to achieve marketing objectives. Moderated or not, usability tests are meant to harvest users’ insights in order to develop an efficient user experience and design an overall better product.
A Beginner’s Guide to Usability & User Testing
In a web marketing context, the process of improving user experience for visitors to a website. While it is part of the design process, UX is more focused on usability and user satisfaction, rather than pure aesthetics. Commonly associated with CRO, UX optimization is more about improving the usability of a site or product for the user, whereas CRO focuses on guiding website visitors to take a desired action.
Value proposition is a succinct presentation of the defining aspects of a product or service that appeal to the customer. Sometimes referred to as a Unique Selling Point (USP), the value proposition should highlight a companies unique advantages in comparison to their competition.
How to Write a Value Proposition
Website optimization encompasses all the technical and marketing techniques used to acquire traffic, engage visitors and then convert them into prospects or buyers using a clearly defined method which meets a number of pre-defined goals.
What Lies Behind Website Optimization?
A subset of personalization, website personalization involves modifying a website in real time in order to match the site’s attributes with the specific needs and characteristics of its visitors, in order to bring them along the conversion funnel with more relevant messaging. This might include creating landing pages that match the sources of traffic, home page headlines that are personalized to a website visitor’s geographic location or time zone, banners or pop-ins that refer to the value of the products they have in their basket, etc.