6min read

Trial and Error of Building a Culture of Experimentation in Marketing | Rand Fishkin

Rand Fishkin discusses the importance of “non-attributable” marketing and why companies should take more risks and allow themselves the freedom to fail.

Rand Fishkin is the co-founder and CEO of SparkToro, a software company that specializes in audience research for targeted marketing. Previously, Rand was the co-founder and CEO of Moz, where he started SEOmoz as a blog that turned into a consulting company, then a software business. Over his seven years as CEO, Rand grew the company to 130+ employees, $30M+ in revenue, and brought website traffic to 30M+ visitors/year. 

He’s also dedicated his professional life to helping people do better marketing through his writing, videos, speaking, and his latest book, Lost and Founder.

AB Tasty’s VP Marketing Marylin Montoya spoke with Rand Fishkin about the culture of experimentation and fear of failure when it comes to marketing channels and investments. Rand also shares some of his recommendations on how to get your brand in front of the right audience. 

Here are some key takeaways from their conversation.

Taking a more risk-based approach

Rand believes there’s too much focus on large markets that people often overlook the enormous potential of smaller markets to go down the more typical venture path. In that sense, founders become biased towards huge, totally addressable markets.

“They don’t consider: here’s this tiny group of people. Maybe there are only three or 4000 people or companies who really need this product, but if I make it for them, they’re going to love it. I think that there’s a tremendous amount of opportunity there. If folks would get out of their head that you have to look for a big market,” Rand says.

People avoid such opportunities because of the regulatory challenges, restrictions, and other barriers to entry that often come with them but for Rand, these underserved markets are worth the risk because competition is scarce. There’s a real potential to build something truly special for those willing to overcome the challenges that come with it, Rand argues. 

There are a lot of underserved niches and many business opportunities out there in the tech world, if companies would shift away from the “growth at all cost” mentality. 

“The thing about being profitable is once you’re there, no one can take the business from you. You can just keep iterating and finding that market, finding new customers, finding new opportunities. But if you are constantly trying to chase growth unprofitably and get to the metrics needed for your next round, you know all that goes out the window,” Rand says.

Freedom to fail

Similarly, Rand states that there’s a huge competitive advantage in committing resources toward marketing channels where attribution is hard or impossible because no one else is investing in these kinds of channels. That’s where Rand believes companies should allocate their resources.

“If you take the worst 10 or 20%, worst performing 10 or 20% of your ads budget, your performance budget, and you shift that over to hard-to-measure, experimental, serendipitous, long-term brand investment types of channels, you are going to see extraordinary results.”

However, the problem is getting buy-in from more senior stakeholders within a company because of these “hard-to-attribute” and “hard-to-measure” channels. In other words, they refuse to invest in channels where they can’t prove an attribute – a change of conversion rate or sales – or return on investment. Thus, any channels that are poor at providing proof of attribution get underinvested. Rand strongly believes that it’s still possible to get clicks on an organic listing of your website and get conversions even if a brand doesn’t spend anything on ads. 

“I think brand and PR and content and social and search and all these other organic things are a huge part of it. But ads are where those companies can charge because the CEO, CMO, CFO haven’t figured out that believing in hard-to-measure channels and hard-to-attribute channels and putting some of your budget towards experimental stuff is the right way to do things,” Rand argues.

According to Rand, these are exactly the kinds of channels where more resources need to be allocated as they generate a higher return on investment than any ad a company might spend on the more typical and bigger name platforms. 

“Your job is to go find the places your audience pays attention to and figure out what your brand could do to be present in those places and recommended by the people who own those channels.”

According to Rand, there is a learning curve in finding the message that resonates with this audience and the content that drives their interest as well as the platforms where you can connect with them and this will all depend on who your audience is.

Experiment with AI

For Rand, the AI boom is more realistic and interesting than previous big tech trends. He especially sees its biggest advantage in solving big problems within organizations that can be best solved with large language model generative AI. 

However, it’s important not to insert AI in a business or create problems just for the sake of using it or to apply it to the wrong places.

“If you find that stuff fascinating and you want to experiment with it and learn more about it, that’s great. I think that’s an awesome thing to do. Just don’t don’t go trying to create problems just to solve this, to use it.” 

He believes the best use case for AI is for more tedious jobs that would be otherwise too time-consuming as opposed to using it for more tactical or strategic marketing advice. Nonetheless, he does believe that there are a lot of interesting and useful solutions and products being built with AI that will solve many problems.

What else can you learn from our conversation with Rand Fishkin?

  • The importance of brand and long-term brand investments
  • Why it’s hard to get leadership to shift away from common ad platforms
  • How social networks have become “closed networks”
  • Why attention needs to shift to your audience and how they can become “recommenders” of your product

About Rand Fishkin

Rand Fishkin is the co-founder and CEO of SparkToro, makers of fine audience research software to make audience research accessible to everyone. He’s also the founder and former CEO of Moz and also co-founded alongside Dharmesh Shah, which was sold to Hubspot in 2014. Rand has become a frequent worldwide keynote speaker over the years on marketing and entrepreneurship with a mission to help people do better marketing. 

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Subscribe to
our Newsletter

bloc Newsletter EN

We will process and store your personal data to respond to send you communications as described in our  Privacy Policy.


7min read

Taking an Outcome-Driven Approach | Ruben de Boer

Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide

Ruben de Boer is a lead CRO Manager and consultant with over 14 years of experience in data and optimization. At Online Dialogue, Ruben leads the Conversion Managers team, developing team skills and quality as well as setting the team strategy and goals. He spreads his knowledge far both as a teacher with Udemy with over 12,000 students and as a public speaker on topics such as experimentation, change management, CRO and personal growth.

In 2019, Ruben founded his company, Conversion Ideas, where he helps people kick start their career in Conversion Rate Optimization and Experimentation by providing affordable, high-quality online courses and a number of resources.

AB Tasty’s VP Marketing Marylin Montoya spoke with Ruben about exciting trends and evolutions within the world of experimentation, including  the various ways AI can impact the optimization of the experimentation process. Ruben also shares ways to involve cross-functional teams to implement a successful culture of experimentation within the organization and why it’s important to steer these teams towards an outcome- rather than an output-driven mindset.

Here are some key takeaways from their conversation. 

The goal should always be outcome-driven

Based on his experience, Ruben believes that one of the biggest pitfalls companies face when trying to kick start their experimentation journey is they focus more on outputs rather than outcomes.

“When a company is still very much in an output mindset, meaning we have to deliver an X amount of sprint points per sprint and we have to release so many new features this year, then of course experimentation can be seen as something that slows it down, right?  Let’s say as a rule of thumb, 25% of A/B tests or experiments result in a winner and so 75% of what was built will not be released, which means the manager does not get the output goals.”

In this scenario, experimentation becomes an obstacle that slows down these outputs. Whereas, when a company shifts towards an outcome mindset, it makes more sense to run experiments with the goal to create more value for the customer. With an outcome-mindset, teams embrace experimentation with customers at the heart of the process.

When teams are more outcome-oriented, the product is based more on research and experiments instead of a fixed long-term roadmap. According to Ruben, it’s vital that companies adopt such a way of working as it helps create better products and business outcomes, which ultimately helps them maintain their competitive advantage.

Importance of cross-functional teams

Ruben argues that experimentation is maturing in that it’s becoming more embedded within product teams.

He notes there’s a rising trend of different teams working together, which Ruben believes is essential for knowledge sharing when it comes to learning new things about the customer journey and the product itself. For Ruben, this helps create an ideal, healthy experimentation environment for teams to experiment better and get the results they want. 

Ideally, there would be experts in experimentation coming in from different teams sharing knowledge, ideas and insights on a regular basis which helps drive inspiration and innovation when it comes to future test ideas. 

The recipe behind the success of these experimentation teams varies and depends on the maturity of the experimentation program and the skills of these teams.  

This could start with a look into the culture of the organization by sending questionnaires to various teams to understand their work process and how autonomous they are. This analysis would also help teams to understand what their current state of experimentation is like such as how accepting they are of experimentation. This helps to devise a strategy and roadmap to successfully implement a culture of experimentation throughout the whole organization.  

This culture scan also helps determine the maturity of an experimentation program.

“Process, data, team, scope, alignment, and company culture: that’s what I generally look at when I assess the maturity of an organization. Is there a CRO specialist throughout the different product teams? How’s decision-making being done by leadership? Is it based on the HIPPO decisions or fully based on experimentation? Then there’s the outcome versus output mindset, the scope and alignment of experimentation as well as the structure of the team- is it just a single CRO specialist or a multidisciplinary team? What does the process look like? Is it just a single CRO process or is it a process embedded in a project team?” Ruben says.

A world of possibilities with AI

With the advent of AI technology, Ruben believes there’s a lot of possibilities with what can be done with it, particularly in the experimentation process. 

While he admits it’s still too early to speculate and that there are also the many privacy concerns that come with such technology, he believes AI can bring a lot of exciting things in the future.

“It would be so nice to have an AI go over experiments on the product detail page with all the results and all the learnings, and just ask the AI, what did I actually learn and what would be good follow up experiments on that? And that would be enormously interesting to have an AI run through all the experiments in the database,” Ruben says.

Therefore, Ruben admits there are a number of possibilities of what teams can do when it comes to designing experiments and saving time and steps in the experimentation process. 

“And just think about maybe three or four years from now, everyone will just have an AI app on their phone and say, I need to buy this and I will buy it for you. And maybe a website with only AI apps on it to purchase stuff, who knows? And then optimization becomes very different all of a sudden.” 

There’s also significant potential with AI when it comes to changing the way people work as well as provide inspiration and ultimately optimize and bring innovation to the experimentation process.

“Maybe based on all the input we give from chat logs, social media channels, reviews, surveys, we can make the AI behave like a user at some point in the future somewhere, which you then don’t have to run user tests anymore because you just let AI see your website.”

What else can you learn from our conversation with Ruben de Boer?

  • Evolving trends in experimentation 
  • His take on change management to help organizations adopt experimentation
  • His own experiences with building cross-functional teams
  • How to tackle resistance when it comes to experimentation   
About Ruben de Boer

With over 14 years of experience as a lead CRO manager and consultant in data and optimization, Ruben is a two-time winner in the Experimentation Elite Awards 2023 and a best-selling instructor on Udemy with over 12,000 students. He is also a public speaker on topics such as experimentation culture, change management, conversion rate optimization, and personal growth. Today, Ruben is the Lead Conversion Manager responsible for leading the Conversion Managers team, developing team skills and quality, setting the team strategy and goals, and business development.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.