February 1, 2013

Be careful what you conclude

“Assume you’re a scientist and you have a flea with six legs”, the joke goes, “that does what you ask him to do.

“So, ask him to jump, and he jumps. Next, cut off two of his legs and ask him, and he jumps. So, cut off two more legs, ask him to jump, and he jumps. Finally, cut off his last two legs, ask him to jump, and he doesn’t.

Conclusion: the flea has become deaf.”

The point is, evidence is usually compatible with various hypotheses.

Be careful what you conclude

When solving complex, ill-defined, non-immediate (CIDNI) problems, you test hypotheses at two stages: during the diagnostics phase, and during the solution-identification phase and testing these hypotheses requires you to look for evidence.

There are several guiding principles you should follow to maximize your chances to get to the right answer—such as considering several, competing hypotheses to reduce the risk of confirmation bias; or favoring the hypothesis that has the least evidence against it, not the one that has the most evidence compatible with it. A particularly relevant one in this situation is to ensure that you have considered all possible explanations (your hypotheses are independent and collectively exhaustive–ICE and not necessarily MECE) and that you are gathering evidence that help you discriminate against all.

This is hard work, as divergent thinking usually is, but worthwhile—otherwise you might end up with perfectly wrong conclusions.

So in the case of our flea friend, once you’ve discovered that, without legs he doesn’t jump, you might design another experiment before concluding, such as asking him to wink at you (do flea eyes even shut?).