In a previous post, we talked about the importance of looking for disconfirming evidence when testing hypotheses. Indeed, this is vital, but, in some situations, supporting evidence might actually be the one helping you reach a solid conclusion. So, don’t discard it.

In some settings, disconfirming evidence is inconclusive

Wason’s work on the importance of disconfirming evidence has been hugely influential over the past half century, in part because it came roughly at the same time as Popper’s approach to evidence. In the 80s, Klayman and Ha used it to shed light on what is meant by “looking for disconfirming evidence:”“What is critical, then, is not the testing of cases that do not fit your hypothesis, but the testing of cases that are most likely to prove you wrong.

It is very important to distinguish between two different sense of “seeking disconfirmation.” One sense is to examine instances that you predict will not have the target property. The other sense is to examine instances you most expect to falsify, rather than verify, your hypothesis. 

The prescriptions of Popper and Platt emphasize the importance of falsification of the hypothesis, whereas empirical investigations have focused more on the testing of instances outside the hypothesized set.” (pp. 215-216)

They also pinpoint the value of a positive test strategy:

“+testing does not necessarily contradict the goal of seeking falsification.”

This agrees with findings by others (Mynatt et al. and Gorman et al.) that a combination of confirming and disconfirming evidence works best.

Account for your own bias

Klayman and Ha argue that, when testing hypotheses, our default mode is to resort to +testing. They argue that this isn’t so bad since, in some cases, this approach might be the only one that can lead to a solid conclusion. But, as Wason showed, for some types of problems, this approach can also be catastrophic.

The problem is that we seem to be hard wired to resort to a positive test strategy. So you shouldn’t worry too much about doing some of that, because it comes to us naturally. Rather, you should focus on making sure you complement this +testing with searching for enough (and appropriate) disconfirming evidence.

Look for both types of evidence and capture your results in your issue tree

The point is that you should be looking for both types of evidence. Hypothesis maps are useful to capture this evidence and how it relates to each of your hypothesis to help you reach appropriate conclusions.

Here, the value of the map is to help clarify your thinking by organizing how evidence and logic support or oppose your hypotheses. A secondary value is that the map serves as a repository for future reference.

[IMAGE MISSING: confirming-evidence-1024x285.png]
Use your issue tree / issue map to relate evidence to your hypotheses.