Experimentation Drives Success

Building a culture of Experimentation in your startup is probably one of THE MOST important things you can do.

As Amazon founder Jeff Bezos once said, “Our success at Amazon is a function of the number of experiments we run per year, per month, per day.”

Experimentation is the means by which Exponential Organizations (ExOs) make data-driven decisions. Each experiment creates a set of learnings that can be used to improve a product, service, and/or process, or test a new product or service. 

In today’s blog, I’ll share key insights and lessons from Amazon and other successful companies and offer tips on how to build a culture of experimentation in your own company. 

Let’s dive in…

NOTE: Understanding how to turn your business into an ExO and increase your growth and impact is a key component of my year-round Abundance360 leadership program. 

Amazon & Experimentation

My friend Jeff Holden is a renowned innovator and past XPRIZE board member. His illustrious career is marked by key roles at Amazon, Uber, and his current venture, Atomic Machines.

As Chief Product Officer at both Amazon and Uber, he spearheaded the creation of Amazon Prime and popular Uber services like Uber Eats and Uber Pool (now UberX Share).

He has a wealth of experience in disruptive innovations, which he distilled down to four key lessons when I sought his advice on an early podcast.

Jeff’s first directive is to design your company as a potent experimental engine from the beginning. In an era where change is the only constant, continuous innovation and Experimentation are crucial for survival and hyper-growth. Jeff incorporated this ideology at Amazon, Groupon, and Uber.

Integrating this experimental engine isn’t an effortless task, and it often requires a cultural transformation in established companies. It also requires a commitment to ceaselessly test bold ideas, new business models, products, and processes. This principle was evident in Amazon’s early days when a comprehensive experimental platform was made available to almost everyone.

This openness led to an onslaught of experiments, many of which proved futile. As Jeff explains, Amazon ran countless experiments out of sheer curiosity, despite their costs and inefficiencies. Eventually, Amazon formed an Experiments Group. Before executing an experiment, its hypothesis and potential value to the company had to be convincingly presented.

Proper interpretation of experimental results is Jeff’s second key lesson. Like Uber, which runs thousands of monthly experiments and bases decisions on statistical significance, knowing the difference between significant and insignificant results is crucial. At Uber, only about 20 to 30% of experiments are successful, but all provide valuable insights. Jeff’s advice? Ignore the external noise, stay focused, and continue to build.

In his third directive, Jeff provides two key pointers about constructing a team and a company:

• Nurture an experimental ethos within the organization and;
• Hire individuals familiar with Experimentation and a data-informed mindset.

Finally, Jeff emphasizes the importance of comfort with being misunderstood in his fourth directive. Strong experimental cultures often invite misunderstanding from external observers.

Jeff cites Amazon Prime, which was initially met with skepticism but now has over 50 million members, as an example: “Amazon Prime could have been one of those catastrophic failures,” says Jeff. “We tried auctions, and that failed, and we tried zShops and that failed. But we just kept going, and we finally cracked it. Then, when we launched it to the world, the response was: ‘You guys are insane! This is like, super risky. You’re going to blow up with all this margin from shipping.’ Bezos, characteristically, replied, ‘Yeah, I kind of figured this would be misunderstood.’”

Key Lessons

As Salim Ismail, my co-author of Exponential Organizations 2.0, and I reflected on Jeff’s directives as well as our own experience, we arrived at several observations and conclusions about the role of experimentation in business.

In this section, I’ll share a summary of these key lessons.

Lesson #1: A to B, then to C

A key insight from the Lean Startup movement is that if you are at point A, you cannot see point C, so focus on getting from A to B.

Once you’re at B, then you can see point C and plot a course toward it. For example, if you’re building a product with 15 features, don’t build them all. Build the two that will get you market traction and feedback from users, then plan the implementation of the rest. This is also known as the MVP or Minimum Viable Product. Experimentation serves as the tool of choice when organizations have a clear destination in mind but lack a predefined roadmap to reach it.

By embracing Experimentation, companies open themselves up to agile and iterative processes that enable them to navigate uncertainty and adapt to changing circumstances.

Lesson #2: Imagination and Experimentation, not Expertise and Experience

A prevailing myth that must be debunked is that an expert holds the key to guaranteed success. As I often say, “The expert is someone who can tell you exactly how something will not work.”

An ExO acknowledges that expertise alone is not sufficient and that relentless Experimentation is essential for progress. By shifting the focus from seeking expert opinions to creating a culture of learning and experimentation, organizations can tap into the collective intelligence of their teams and communities to explore new frontiers.

Salim and I interviewed Sebastian Thrun of Udacity about this topic. Sebastian said, “When hiring, I look for imagination, not experience.”

Lesson #3: Secrecy

Unless Experimentation is the default policy throughout an organization, secrecy is critical when conducting experiments.

Smart organizations often operate in stealth mode to protect against larger market consequences and media exposure when shutting down unsuccessful projects. This freedom to fail in private allows them to experiment more boldly, explore unconventional ideas, and iterate without fear of external judgment.

Lesson #4: A Culture of Learning

An ExO is the embodiment of a “learning organization.”

Creating a culture that encourages taking risks is vital for unleashing the full potential of Experimentation within an organization. By instilling a sense of psychological safety and removing the fear of failure, teams become more inclined to explore uncharted territories.

The best ExOs go beyond mere lip service and establish reward mechanisms, including the use of crypto rewards, to incentivize and recognize successful experiments conducted within the community. Top ExOs give awards and recognition to team members who take big operational risks, fail, and then share their learnings.

Lesson #5: Big Companies Can Also Experiment

Large, mature organizations often face challenges when it comes to adopting a culture of Experimentation: namely higher costs, regulatory barriers, and a fear of failure.

Double that for government departments. However, some multinational corporations have recognized the value of running experiments in smaller markets to mitigate these challenges and test ideas more efficiently. This approach allows them to reduce costs, gather valuable insights, and make informed decisions before scaling globally.

Lesson #6: Living on the Edge

While the above heading sounds dangerous, it’s actually the opposite. By constantly experimenting, organizations stay closer to market realities.

This is especially important in volatile and uncertain times. Their constant learning delivers an agility that enables them to remain responsive to market shifts and customer demands. By embracing Experimentation as a core attribute, ExOs ensure that they are constantly learning and evolving, positioning themselves as frontrunners in their industries.

How to Build a Culture of Experimentation

So, how can you create a culture of experimentation in your business?

If you’re a startup, it’s relatively trivial to implement an experimental culture. This task, however, becomes much more difficult as organizations get bigger. Large organizations are geared toward efficiency and predictability, which are antithetical to experiments that are counter to both those objectives.

Here are some steps to integrate this Experimentation into an organization:

• Appoint a “head of experiments” to educate the team on the qualities of a good experiment and, in turn, prompt continuous Experimentation.

• Award those teams that run experiments and learn, whether they succeed or fail.

• Elevate the Experimentation conversation to your weekly or monthly executive staff meeting. Have your teams report on the experiments they ran and what they learned. Again, reward the effort, but don’t penalize the failures.

• Encourage your management team to ask great questions. This is an analog of the MTP. ExOs think big in almost everything they do—and never more so when experimenting with new products, markets, technologies, organizations, customer experiences, and employee culture. Formulating great questions is an art form. It also takes courage. But somewhere out there is that one question that can lead to a billion-dollar opportunity for your company.

• Utilize generative AI as a thought partner to recommend experiments you should be running, how to design them, and what parameters to measure.

• Run an incentive prize (or series of prizes) in your company to canvas crazy ideas. As described in my book BOLD, MIT’s Michael Schrage said, “The best way to become innovative is to innovate more.” Schrage suggests running a 5x5x5 design competition within companies to encourage such innovation. Here’s how that works: 
  • Step 1: Within your company or group, organize a minimum of 5 teams of 5 people each. The teams should have a diverse makeup of executives, engineers, marketing members, executive assistants, customers, and suppliers.
  • Step 2: Give each team a period of 5 days to come up with a portfolio of 5 “business experiments” that should take no longer than 5 weeks to run and cost no more than $5,000 to conduct. Each experiment should have a business case that explains how running that experiment will give tremendous insight into a possible savings or growth of $5 million for the firm. Tell the teams that you’re not looking for incremental improvement but significant changes that are exponential in nature.
  • Step 3: Run the 5-week experiments and have each team report on their outcome. If one or two of the experiments yields significant results, invest in those ideas to develop new products or services.

Why This Matters

In today’s dynamic business landscape, adaptability and innovation are crucial.

Experimentation is a tool of transformative potential. It goes beyond mere testing and trial and error—it’s about cultivating a mindset of discovery, driving innovation, enhancing performance, and embracing the unknown.

As I often say, “The day before a major breakthrough, it’s a crazy idea.”

If you’re not experimenting, then you’re an incremental organization and not an ExO.

A successful culture of Experimentation requires a strong element of Autonomy, the next ExO attribute and the topic of our next blog in this series.