Defining Hypotheses

Startup founders must engage in a lot of experimental learning to find and confirm their markets. The vision matters, of course, but it also must survive its test in the market. To that end, formulating good experiments is essential. The heart of an experiment is the hypothesis it tests. What do you expect to happen? How do you measure the outcome? What is your threshold for success? A poorly defined hypothesis clouds the outcome and impedes the search for clear signal. So here are two pieces of advice for startup founders on how to formulate a good hypothesis.

First, make it boolean. Make the hypothesis yield a pass/fail result. This is the simplest possible set of outcomes. Second, include your criteria for success or failure in the hypothesis.

These two points are important for the same reason. If you apply each of them and then run your experiment, you will have a clear outcome: we passed or failed according to criteria established at the outset. With that clarity, you will discuss what it means that you passed or failed. This is the forward-looking question you need to address. It focuses on how you will behave in light of the result. What changes you can make your hypothesis? How does the result alter your vision? What strategy you should adopt in response to this new finding? And so on.

In contrast, if you do not establish criteria beforehand, or if you produce a complicated set of outcomes, then you will discuss whether or not you passed or failed. That is a backward looking question, one typically loaded with emotion from wanting to succeed by passing. It does not focus on options for reacting to the outcome, and instead obsesses over the outcome itself.

This is simple and yet in practice proves difficult. I’ve seen more hypotheses that violate these two rules than ones that adhere to them. Defering the formulation of clear success criteria hedges your bet. You can adjust them later so that your experiment succeeded by passing. That generally makes people feel good because they can claim to have been right all along. But it involves a mistaken notion of experimental success.

The success of an experiment hinges on the learning that it provides. Both pass and fail outcomes support learning. There is no reason to believe that you learn more from “pass” than from “fail” when running an experiment. If all of your experiments succeed, you are playing it too safe and not stretching far enough into the unknown. If all of your experiments fail, then you are out of touch with your own capabilities. A good rule of thumb is to aim for a 50% success rate. If you read “pass” as “success” then that sounds abysmal. But both “pass” and “fail” contribute to experimental success.

Finding the right thresholds to apply in your pass/fail criteria is not always easy. You are dealing with unknowns when learning, so of course you will miss the mark with some frequency. With success criteria defined in advance you can always decide that the failure was due to erroneous criteria and formulate a new hypothesis. But if you establish the criteria after the fact you will have lost the clear signal from your experiment. That may feel better, but it does less to advance your understanding of the market, and diminishes the objectivity of the outcome. Market understanding matters far more than your feelings.

Over time as a product matures, the experiments will naturally trend toward changes of lesser consequence. You don’t want to make a major change to a deployed system that fails your user 50% of the time. But that’s not the same question as pass/fail of your experiment. Instead of changing the pass rate in your experiments, reduce the scope of impact. Entrepreneurs usually have experience working in larger companies. And they often bring with them the practice of making narrow-scope changes and conflating pass with success. In an immature business you do not have time for that. Scope your experiment appropriate to the risk of failing to learn.

In summary, restrict yourself to pass/fail results, and agree on your success criteria before running your experiment. Doing so you will give you the clarity needed to focus forward and adapt.