This year, my friend Simon has been working a Mastermind game for the Blackberry, and when I went back to Boston I got a chance to try it out. It looks great, but I was actually much more impressed at how good he’d become at the game after thousands of testing sessions. I took quite a few guesses to finish games with 4 and 5 secret numbers, whereas he has been able to finish games with 9 slots in only 13 guesses!
When I asked Simon what his secret was, he told me that you have to make each line teach you something you can be sure about. You can’t just guess randomly, and you can’t try to learn too much at once. When Simon plays, each line is an experiment; he knows what every possible result will mean to him before he submits it.
Not only is this great advice for a game of Mastermind, it’s great advice for a startup. Lean startups are all about feedback, but it’s very easy to start looking at your Google Analytics reports without knowing what they’re really telling you and how you should react to them. Now, I have a spreadsheet on which I write down any hypotheses that I have, a test that will allow me to test the hypothesis, and what the next step is if the test confirms or denies my hypothesis. The actual column names are ‘Hypothesis’, ‘Test Idea’, ‘Required Results to Confirm’, ‘Required Results to Disprove’, ‘Action if Confirmed’, ‘Action if Disproved’, ‘Real Test’, ‘Results’ (these last two are filled in over time as we run the tests).
I’ve only just started doing this, so I only have one completed test so far, but it’s a good reminder to think critically about what exactly I learn from any action I take, and I hope it will help Bespoke Row improve quickly in the future. If I get enough comments here, I’ll write a follow up in three months or so on how it works out.


