A few weeks ago, we talked about the importance of looking for disconfirming evidence when testing hypotheses. Indeed, this is vital, but, in some situations, supporting evidence might actually be the one helping you reach a solid conclusion. So, don’t discard it.
In some settings, disconfirming evidence is inconclusive
Bazerman and Neale, which we discussed in our previous post, reported Wason’s work that disconfirming evidence was the critical one. Wason’s work has been hugely influential over the past half century, in part because it came roughly at the same time as Popper’s approach to evidence. In the 80s, Klayman and Ha used it to shed light on what is meant by “looking for disconfirming evidence:”
“What is critical, then, is not the testing of cases that do not fit your hypothesis, but the testing of cases that are most likely to prove you wrong. […]
It is very important to distinguish between two different sense of “seeking disconfirmation.” One sense is to examine instances that you predict will not have the target property. The other sense is to examine instances you most expect to falsify, rather than verify, your hypothesis. […]
The prescriptions of Popper and Platt emphasize the importance of falsification of the hypothesis, whereas empirical investigations have focused more on the testing of instances outside the hypothesized set.” (pp. 215-216)
They also pinpoint the value of a positive test strategy:
“+testing [i.e., testing hypotheses by looking at instances where the target property is hypothesized to be present] does not necessarily contradict the goal of seeking falsification.”
This agrees with findings by others (Mynatt et al. and Gorman et al.) that a combination of confirming and disconfirming evidence works best.
Account for your own bias
Klayman and Ha argue that, when testing hypotheses, our default mode is to resort to +testing. They argue that this isn’t so bad since, in some cases, this approach might be the only one that can lead to a solid conclusion. But, as Wason showed, for some types of problems, this approach can also be catastrophic.
The problem is that we seem to be hard wired to resort to positive test strategy. So you shouldn’t worry too much about doing some of that, because it comes to us naturally. Rather, you should focus on making sure you complement this +testing with searching for enough (and the right) disconfirming evidence.
Look for both types of evidence and capture your results in your issue tree
The point is that you should be looking for both types. Issue trees are useful to capture this evidence and how it relates to each of your hypothesis to help you reach appropriate conclusions.
Here, the value of the tree is to help clarify your thinking by organizing how evidence and logic support or object your hypotheses. A secondary value is that the tree serves as a repository for future reference.
Bazerman, M. H. and M. A. Neale (1992). Negotiating rationally. New York, The Free Press.
Gorman, M. E., et al. (1984). “How disconfirmatory, confirmatory and combined strategies affect group problem solving.” British journal of psychology 75(1): 65-79.
Klayman, J. and Y.-W. Ha (1987). “Confirmation, disconfirmation, and information in hypothesis testing.” Psychological review 94(2): 211.
Mynatt, C. R., et al. (1978). “Consequences of confirmation and disconfirmation in a simulated research environment.” The Quarterly Journal of Experimental Psychology 30(3): 395-406.
Wason, P. C. (1960). “On the failure to eliminate hypotheses in a conceptual task.” Quarterly journal of experimental psychology 12(3): 129-140.