Bazerman and Neale, in their excellent Negotiating rationally, cited Wason's famed problem:Here is a three-number sequence: 2–4–6. Your task is to discover the numeric rule that produced these numbers. To determine the rule, you can generate other sets of three numbers that we will acknowledge as either conforming or not conforming to the actual rule. You can stop producing sets of three numbers when you think you've discovered the rule. How would go about this task?

Take a moment to generate a few of those sets.

Popular guesses, they report, include "4–6–8" and "10–12–14", that is, "ascending even numbers." That rule (hypothesis) is consistent with the given sequence (evidence) but it isn't the rule they have in mind.

A popular second guess is "5–10–15" or "100–200–300," which obeys "the difference between the first two numbers equals the difference between the last two numbers." Again, kudos for the effort, but wrong guess.

The rule that Wason had in mind is "any three ascending numbers." And to get to it, you need to accumulate disconfirming evidence.

Often, the best evidence is disconfirming

One big problem with evidence is that it's usually compatible with more than one hypothesis. So, it is not because you found evidence compatible with your hypothesis that your hypothesis is correct.

Let's look at this graphically in a Venn diagram. "Any three ascending numbers" would be the largest bubble (although we could imagine larger ones, still, such as "any three numbers" or even "any sequence of more than one number," which are also compatible with the " 2–4–6" set). That first rule would include "any three ascending even numbers," which is, clearly, a subset. Likewise, "any three numbers where the difference between the first two numbers equals that between the last two" is also a subset. And we could come up with other rules, for instance "any three numbers where the third equals the sum of the first two." In the end, we might end up with the figure below.

[IMAGE MISSING: Evidence-is-usually-compatible-with-more-than-one-hypothesis-1024x538.png]
Evidence is usually compatible with more than one hypothesis

Now, our  sequence "2–4–6" is compatible with these four rules. To find out which rule in particular governs it, we shouldn't look at sets that conform with these four rules but, rather, that violate one or more.

If you propose the sequence " 2–12–100," and you hear that it conforms with the rule, you are making significant progress because it allows you to rule out to of our four candidate rule: rule #3 would yield "2–12–22" and rule #4 would yield "2–12–14."

[IMAGE MISSING: Look-for-disconfirming-evidence-1024x535.png]
In some settings, such as Wason's 2-4-6 problem, looking for disconfirming evidence is the only way to reach the right conclusion

You can continue this process of elimination by proposing, for instance, "2–13–100," a sequence that conforms with only one of our remaining two candidate rules. If you hear it conforms with the rule, you can rule out proposal #2.

There is nothing new here. Indeed, any mathematician will tell you that to show that a theorem is wrong, all you have to do is find one counter example. Or in Taleb's words: It is not because you have seen a hundred swans and all of them were white that you should conclude that all swans are white. However, it only takes you seeing one black swan to be able to conclude that not all swans are white.However, looking for disconfirming evidence doesn't come naturally to us. (As a test, think about your default news source; is it generally aligned with your political views or generally opposed to them? Yep, welcome to the club.)The take away is simple: When it comes to hypothesis testing, aim at shooting down hypotheses. Attack all of them energetically—particularly your favorite one—and then take a step back to update your thinking. The one that best weathers this critical-thinking attack should be your new favorite one.

References:

Bazerman, M. H. and M. A. Neale (1992). Negotiating rationally. New York, The Free Press, p. 62.Chevallier, A. (2016). Strategic Thinking in Complex Problem Solving. Oxford, UK, Oxford University Press, pp. 91–102.Taleb, N. N. (2012). Antifragile: things that gain from disorder, Random House LLC.Wason, P. C. (1960). “On the failure to eliminate hypotheses in a conceptual task.” Quarterly journal of experimental psychology 
12(3): 129-140.Image credit: Pixabay.