I have portfolio of lean startup ideas or one startup idea but several versions for its lean canvas. And I have a limited capacity for testing these ideas. I can’t test all of them simultaneously and I wonder : which one should I start testing first ? which lean canvas should I start with ?

First, let’s remember why we always start testing the most uncertain hypothesis when given one canvas. For your startup to be a success, all of the hypotheses in the canvas have to be tested and validated as true. Hence the cost of testing the whole canvas : it is the sum of the costs for testing each and every of its hypotheses. You test the canvas and test it further and cumulate testing costs. But if ever the hypothesis you just tested is false, you have to pivot and start again with a modified canvas, a new version of your canvas, or you have to put that project aside and further test another one. You therefore hope you start by testing a false hypothesis rather than a true one. You’d rather spend as low as possible before invalidating the whole canvas. By failing fast, you limit the cost of testing the whole canvas and you can start with an hopefully better version or give up with this project. In other words, you should try to pick the hypothesis which has the highest probability of being false. That’s the most uncertain hypothesis.

But one has to notice that some tests are expensive while others are cheap. How to take the cost of testing into account ? You should multiply the probability of each hypothesis with the cost required for testing it. You then pick the hypothesis which has the lowest product of probability and testing cost. Given a lean canvas, you can annotate your hypotheses :

• P=1 for low probability hypotheses, P=2 for medium probability, P=3 for high probability,
• C=1 for low cost tests, C=2 for medium cost tests, C=3 for most expensive tests

You then multiply P and C and get P*C products from P*C=1*1 =1 to P*C=3*3=9 and start with the lowest P*C hypothesis.

Now, let’s get back to my initial problem. I have a portfolio of lean startup ideas and have to pick the first one to be tested. Which idea should I pay the most attention to first ?

Let’s calculate the probable cost for testing a whole canvas. This cost is not the sum of the cost of each and every hypothesis in this canvas because I may have to abandon this canvas as soon as one of its hypotheses is proven to be false.

Let’s call :

• H1, H2, …, H9 the 9 hypotheses in this canvas, in their order of testing (lowest P*C first)
• p(H1), p(H2), …, p(H9) the probabilities of these hypotheses (0 <= p(Hi) <= 1)
• c(H1), c(H2), …, c(H9) the cost for testing these hypotheses
• C the probable cost of testing this whole lean canvas of hypotheses

At first, there is the cost of testing H1. If H1 is found to be true, you then have the cost of testing H2. But we don’t know beforehand if H1 is true or false. We have to take the probability of H1 into account before adding the cost of testing H2. So the cost of testing one whole canvas is :

C = 1*c(H1) + p(H1)*c(H2) + p(H1)*p(H2)*C(H3) + … + p(H1)*p(H2)*…*p(H8)*c(H9)

Let’s assume that all of these costs have the same order of magnitude. Then the terms on the far right of this sum are negligible compared to the first terms. In other words, you don’t have to calculate all of these items and you can approximate the cost of testing the whole canvas :

C =~ c(H1) + p(H1)*c(H2)

or even, assuming that p(H1) =~ 0 (it’s the most uncertain hypothesis) and c(H2) =~ c(H1) (they are of a same order of magnitude), you can further approximate :

C =~ c(H1)

So the cost for testing a whole canvas is approximately equal to the cost for testing its most uncertain hypothesis, or its hypothesis with the lowest product of cost and probability.

Now I have my collection of lean canvases and I can calculate the probable cost of their testing :

• for each canvas, I have to identify its hypothesis H1 with the lowest product of cost and probability,
• the probable cost for testing this canvas is approximately the cost of H1

What is my collection made of ? Let’s say it’s a collection of several versions of a same project. On one hand you hope your first attempted canvas will be the right one so that you don’t spend time and effort testing bad versions. On the other hand you have to take the cost of testing into account and would prefer cheap tests to expensive ones. So your best version is the one with the best ratio of probability to cost. You want to maximize P / C.

We know C =~c(H1) but what’s the total probability of a canvas ? It’s the product of the probabilities of its hypothesis :

P = p(H1) * p(H2) * … * p(H9)

So :

P / C =~ p(H1) * p(H2) * … * p(H9) / c(H1)

Let’s call :

• version A and version B the versions of canvas to be compared and optimally ordered for testing
• H1A, H2A, …, H9A the hypothesis in version A of the canvas
• H1B, H2B, …, H9B the hypothesis in version B of the canvas
• PA and PB the probabilities of version A and version B
• CA and CB the costs for testing version A and version B

We want to compare the probability to cost ratio of these 2 canvas :

PA/CA =~ p(H1A) * p(H2A) * … * p(H9A) / c(H1A)
PB/CB =~ p(H1B) * p(H2B) * … * p(H9B) / c(H1B)

For this comparison, let’s calculate PA/CA * CB/PB. Is it more or less than 1 ?

Given that these are 2 versions of the same lean startup projects, they probably share a significant number of hypotheses. You can eliminate these hypothesis and only keep this limited number of hypotheses which differ from A to B, which we will call :

• HiA, …, HnA in version A
• HiB, …, HnB in version B

So :

PA/CA * CB/PB =~ ( p(HiA) * … * p(HnA) ) / ( p(HiB) * … * p(HnB) ) * c(H1B) / c(H1A)

If only one hypothesis Hi differs from A to B, you then have to compare p(HiA)/c(H1A) and p(HiB)/c(H1B). In other cases, the product of several probabilities will … probably… be more different from version A to version B than the cost of their first hypothesis (which may even be the same hypothesis) and you can then approximate :

PA/CA * CB/PB =~ ( p(HiA) * … * p(HnA) ) / ( p(HiB) * … * p(HnB) )

In other words, you simply want to start with the most probable version and only take cost into account if you can’t make up your mind considering probabilities only.

You then take the version with the cheapest first test.

Now let’s add one last difficulty : my portfolio is a collection of distinct projects which I want to test in an optimal order. You want to maximize the probability of validating a high value canvas with a minimal testing cost. So you have to estimate :

• V the value of each canvas (assuming they will be proven to be true and successful)
• P their respective probability
• C the cost for testing their first hypothesis (their least probable and cheapest hypothesis)

And you want to pick the project with the highest V*P/C

Assuming that the cost for testing hypotheses is of the same order of magnitude from project to project, you can consider V*P only and pick the project with the highest V*P value (unless your V*P values are similar and one of the first hypotheses has a surprising cost which you then have to take into account).

Let’s summarize… Given a collection of lean startup projects described as canvases. Some projects may have several versions of their canvas. And each canvas is a set of 9 hypothesis. You want to pick the best hypothesis to start testing :

1. First, for those projects with several versions, only consider the most probable version and put the other versions aside
2. Now order your projects by value according to their value assuming they are successful and group them into 3 classes :
   • V=1 for projects with lowest value,
   • V=2 for projects of medium value
   • V=3 of highly valuable projects
3. Do the same with their probability of success in 10 years
   • P=1 for the most risky projects
   • P=2 for mediumly risky projects
   • P=3 for projects with a higher chance of success
4. Pick the project with the highest P*V value. In case of tied projects, restart steps 2 and 3 with these projects only and discriminate them. If you can’t make up your mind, go on with step 5 for each of the remaining projects and pick the one with the cheapest first test.
5. Hopefully you selected one project. Rate the costs and probabilities of its 9 hypotheses, pick the one hypothesis with the lowest product of cost and probability (or further discriminate your best hypotheses by rating them once again).
6. Test this hypothesis.
7. Hopefully your test was a success, meaning that you know your hypothesis was true (you should be surprised given it was the least probable in this canvas) or false (which still means the test was a success: this is validated learning). What to do now ? If the hypothesis is false, this canvas is invalidated and you have to pick another one. If the hypothesis is validated as true, this project just turned even more probable. But maybe your portfolio now contains new projects or this test gave you ideas for even more valuable versions of the canvas of this project ? So get back to step 1 !
8. Finally, what if your test takes a long time before validating or invalidating its hypothesis (e.g. you have to wait for an appointment or for the next season of activity) ? Which hypothesis to test next ? You should get back to step 1 but consider that the hypothesis which is still being tested still has the same probability but its cost is now null (or diminished by the amount of time and effort spent on starting its test and waiting for results). So the probability of its canvas is still the same as before but its cost is approximately the cost for testing its next hypothesis plus the cost for finishing the first test.

That’s it. Now how do YOU proceed practically speaking for ordering your portfolio ? How do you optimize the ordering of tests in your portfolio of hypotheses without wasting time and effort and over-optimizing ?