Innovation 2.0: EXPERIMENTING TO IMPROVE, NOT TO PROVE

by Jean-Louis Barsoux, Cyril Bouquet and Michael Wade

ACCORDING TO RESEARCH, THE TOP REASON start-ups fail is that they offer a product or service that nobody wants. Put simply, they have come up with an idea that is viable, but it is not desirable. To establish whether an idea is both viable and desirable, you must engage in a specific type of experimentation.

The tightrope of experimentation combines two seemingly contradictory qualities: total focus and total flexibility. By contrast, most conventional experimenters are single-mindedly focused on validating their idea. Typically, a scientist, inventor or entrepreneur develops a hypothesis and then sets out to prove it through experimentation.

This emphasis on validation is rooted in the scientific method, which is often held up as the ideal of experimentation. Valid scientific protocols are based on formulating a hypotheses, testing it (with treatment and control groups), analyzing the results, and drawing valid conclusions. Although these protocols help you move away from unscientific gut instincts in search of valid evidence, too much focus on validation can stifle the flexibility and open-mindedness that is required for successful innovation.

The purpose of experimentation should always be twofold: to test an assumption and learn from the testing, so you can decide whether to persevere, pivot — or pull out. Unfortunately, the traditional emphasis on validation often crowds out robust investigation. In this article we will show how important it is to commit to experimenting to improve, rather than experimenting to prove.

False Experiments and Extreme Experimentation

When your aim is to validate a hypothesis, you are unconsciously drawn into a logic of supporting what you already believe to be true, rather than seeking objective answers. Instead of exploring uncharted paths, you look for ways to bolster existing assumptions. You become prone to confirmation biases—and blind to unexpected data.

As a result, you can easily become a prisoner of your initial assumptions, as shown in research on the ‘imprinting effects’ of early decisions during a company’s creation, including the type of people hired. Researchers found that these decisions set a trajectory that dictates subsequent moves and becomes difficult to alter.

When innovators fail to challenge their initial assumptions, the consequences can be dire. The Segway is a case in point. Dean Kamen, inventor of this motorized, self-balancing personal vehicle, predicted it would “be to the car what the car was to the horse and buggy.” He believed Segways would be used everywhere: from theme parks to battlefields to factory floors, and in cities from Seattle to Shanghai. “Cars are great for going long distances, but it makes no sense at all for people in cities to use a 4,000-lb. piece of metal to haul their 150-lb. asses around town,” he said. Kamen envisioned urban centres freed of cars to make room for millions of Segway-powered pedestrians.

The Segway’s self-balancing system was indeed a technological marvel, and Kamen — with several breakthrough inventions under his belt — had no trouble attracting funding. Unfortunately, the entire project was so shrouded in secrecy that the Segway went through three or four iterations based exclusively on internal feedback, rather than user feedback. Concerned that someone would steal his idea, Kamen didn’t seek out enough critical customers.

As indicated, once you start testing your cherished idea, the chief risk is that well-established confirmation biases and sunk-cost effects will deaden your responsiveness to corrective feedback. Like Kamen, you may set up experiments to ratify your assumptions rather than refute them. You may engage in false experiments with unreliable feedback providers — family, friends, or like-minded thinkers — rather than real users. And the more you invest, the harder it may be to abandon the project or refocus your efforts.

Investigation is a more open-ended form of experimentation. It is designed to illuminate overlooked factors or preferences, leaving room for unexpected insights to emerge. Experimenting should never lock you into a cycle. It should provide you with opportunities to go in very different directions. You have to test your offerings earlier, on a smaller scale, and in more hostile environments. And when you receive criticism, you have to take it on board. Even when you are in validation mode, you must preserve space for discovery. Remember, you are testing to improve, not just to prove.

Following are some key principles to embrace.

WELCOME SURPRISES. To preserve your capacity to investigate (make new discoveries and capture opportunities), even as you try to validate assumptions, you must welcome and accept surprises.

By preserving an exploratory mindset, you can acquire faster, richer, and more unexpected data from the outside world. A prime example of such thinking is architect Frank Gehry, whose pioneering designs are the result of a very unorthodox approach to experimentation.

Gehry’s fame stems from the instantly recognizable aesthetic of his buildings. His architectural sensibility was immortalized in an episode of The Simpsons in which Marge writes a letter asking him to build an opera house for her hometown of Springfield. Gehry balls up the letter and tosses it to the ground. But after looking at the crumpled letter, he’s suddenly inspired to build a similar-looking concert hall for Springfield.

Every Gehry building starts with a distinctive sketch. As captured in Sydney Pollack’s classic documentary Sketches of Frank Gehry, his style is fluid but messy. His initial sketches are unconstrained by architectural norms: They are loose, squiggly, and impressionistic rather than precise — a far cry from the clean lines of conventional architectural renderings. Gehry’s sketches pay no heed to practicalities or even gravity, but this impulsive quality prevents him from becoming formulaic or repetitive in his designs. Although the drawings provide only loose directions, in retrospect, you can see that they contain the dynamic energy of the final design. They are intended to convey a feeling that will excite — or sometimes disturb — the client.

The journey from rough sketch to finished building is a long one, and though Gehry quickly translates the sketches into three-dimensional models, there is a prolonged back-and-forth between models and drawings. In other words, the experimentation process remains in a ‘liquid state’ for a long time. One reason for this is that Gehry resists the temptation to settle on the ‘right’ model, preferring instead to create a variety of study models— what he dubs ‘shrek’ (a Yiddish word for ‘fright’) prototypes.

Gehry wants the shrek models to make clients nervous. His aim is to provoke a strong reaction, whether positive or negative, so he can better understand how the client thinks. Based on the client input, he makes refinements and creates new models. These successive models don’t build on the ones that came before. Instead, they represent divergent solutions. This process enables Gehry to continue exploring and learning from the clients’ discomfort. In this way, his ideas gradually ferment and mature over time.

Though he favours unorthodox forms, Gehry is very mindful of function. When the Walt Disney Concert Hall opened in 2003, it was widely praised for its acoustics. Esa-Pekka Salonen, the music director of the Los Angeles Philharmonic, who worked closely with Gehry on the design, commented: “Frank was very clear from the beginning. He said, ‘This is a hall for the orchestra, and this is a building for music. And that has to be the first priority, and everything else is of lesser importance.’ And I thought this was quite a statement from an architect.”

Because his designs are unconventional, Gehry is sometimes misrepresented as someone who imposes designs on clients — take it or leave it. “That’s absolutely the opposite of who he is,” said Vanity Fair’s architecture critic, Paul Goldberger. “He goes through multiple iterations of every project and it’s not only his thinking but he’s very, very eager for feedback from clients and for dialog with them.”

PROPOSE MULTIPLE MODELS. Gehry proposes a multitude of designs. As an experimenter, this gives him two advantages. First, it keeps him from getting too attached to any one idea. You can more easily remain open to learning — and pivoting — if you are not overinvested in your prototype. The longer you pursue an idea, the more you feel as though your reputation is tied to it.

As Gehry’s colleague, Jim Glymph, put it: “If you freeze an idea too quickly, you fall in love with it. If you refine it too quickly, you become attached to it. And it becomes very hard to keep exploring, to keep looking for better. So, the crudeness of the early models, in particular, is very deliberate. It keeps you from bonding with the idea so much that you can’t move on.”

Second, it speeds discovery and lets you pivot earlier, before hitting a wall. The lean start-up approach is very much sequential. You set off in one direction, go through iterative cycles of buildmeasure-learn, pivot when an assumption is clearly refuted, and progressively home in on a product-market fit. A more radical approach is to create multiple concepts and pursue the option that seems to have the most traction with future users, as shown by page views, sign-ups, or presales.

The primary aim is to get feedback and a first impression of whether the proposal resonates with independent experts or actual paying customers. Market testing in parallel is all the easier in the digital realm, where it’s fast and cheap to realistically simulate nonexistent products or services through landing pages. This way, no opportunity is lost or inadvertently dismissed.

PROVOKE STRONG REACTIONS. Gehry doesn’t just propose several sketches or models. He develops extreme points of view that provoke strong emotional reactions. Why? Because he values negative feedback as much as positive. He uses the shrek models to test people at the boundaries, making them ill at ease so they will react.

One professor involved in the committee overseeing the construction of the Peter B. Lewis building at Case Western University recalled the experience of working with Frank Gehry: “He often would say about a model he was presenting to the university team: This is not what we are doing. And it was difficult to appreciate what he meant until we followed the design as it evolved through dozens of iterations.”

The crude sketch of the early model was just a means of getting a reaction, exploring possibilities and investigating issues relevant to the building project. Though his fluid sketches are short on detail, they capture the energy and spirit of the proposed vision. And because his alternative models don’t build on each other, as if they were presented in sequence, Gehry is able to constantly surprise clients and maintain their discomfort to encourage a steady stream of feedback throughout the design process. Gehry is not interested in knowing what the building will look like at the very start. He’s willing to engage in open and vibrant discussions with clients so that together they can find out.

Beyond asking the right questions, Gehry shows a willingness to heed the answers and integrate them into his thinking.

ACCEPT AND INCORPORATE FEEDBACK. Eliciting rich data is one thing, but once you get those inputs, how do you make sense of them? And are you prepared to deal with the implications? Even if you acquire robust feedback through parallel testing or provocation, you could fail if you aren’t receptive enough to the findings.

We know from the Segway case that a passion for your idea can get in the way of hearing constructive and well-intentioned feedback. So how can entrepreneurs stay objective? After all, the passion required for entrepreneurship often stems from your love affair with your vision or creation. You must not only invite feedback; you must also accept and leverage it.

There are two key facets of accepting feedback. Whether you’re working alone or in a team, you need to think about how you interpret incoming data and how to avoid biases. There are two ways to address these challenges: (1) let the data speak, and (2) seek out people who think differently than you.

LET THE DATA SPEAK. At Pixar, co-founder Ed Catmull oversaw a parade of film directors who were trying to bring their creative projects to the screen. During this time, he noticed systematic patterns of resistance to constructive and well-intentioned feed- back. At some point in the life of a project, the director’s passion would blind them to the movie’s inevitable problems.36

In his book Creativity, Inc., Catmull recalled directors’ reactions when receiving feedback from senior colleagues. When the stakes are high and people in the room don’t seem to understand a director’s vision, he said, “it can feel to that director like everything they’ve worked so hard on is in jeopardy, under attack. Their brains go into overdrive, reading all of the subtexts and fighting off the perceived threats to what they’ve built.”

Although the feedback might pinpoint what’s wrong or missing, candid and empathic feedback is only valuable if “the person on the receiving end [is] open to it and willing, if necessary, to let go of the things that don’t work.” Catmull stressed that the best feedback “can’t help people who refuse to hear criticism without getting defensive, or who don’t have the talent to digest feedback, reset, and start again.”

By contrast, one key to Frank Gehry’s success has been his ability to admit failure — to reject his initial attempt (in part or altogether) and start over with renewed confidence that the next draft would be better because of what he has learned. With one wealthy client, Peter Lewis, he worked for 12 years in order to