Deinonychus antirrhopus

Decting Design: Elimination vs. Comparison

"How are design hypotheses properly inferred, simply by eliminating chance hypotheses or by comparing the likelihood of chance and design hypotheses?"

This is a question that William Dembski has asked and tried to answer. Currently, most philosophers of science prefer to use the Bayesian approach. The Bayesian approach can be seen as an outgrowth of the Likelihood Principle. This approach is essentially comparative. That is, we decide on which hypothesis (theory) to use based on a comparisons of the probabilities for each hypothesis being true, given the data (evidence). William Dembski has tried to argue that this approach is insufficient when it comes to design vs. chance hypotheses.

To illustrate the problems he gives us two "simple" examples to highlight how the Bayesian and what he calls the Fisherian (I'd call this the Frequentist or Classical, but whatever) approaches to (statistical) inference work. The Fisherian example is the simpler of the two. With the Fisherian approach we specify a rejection region, say 1%, and if the data fall within that region then we reject the chance hypothesis. To see how it works, Dembski uses and example with a coin. The coin could be fair (the chance hypothesis) or unfair (the design hypothesis). If fair the probability of getting heads is 0.5, and if unfair the probability of heads is 0.9 (don't ask me how one represents chance and the other design when both involve chance--i.e. uncertainty in the outcome). Now, if we flip the coin 10 times and get 10 heads we note that if the coin is fair we'd reject this hypothesis at the 1% level because if the coin is fair such an outcome occurs only .01% of the time. That is the data is within the rejection region so we reject the chance hypothesis. So far, so good.

Now with the Bayesian example, Dembski makes things a bit more complicated. First, we have two coins. One is fair the other unfair (with the same probabilities as above in regards to obtaining heads). But, before we flip we must make a draw from an urn. In the urn are 1 million equally sized balls, with all but one being white and the one exception being black. Now, if we draw the black ball we flip the unfair coin, and if we draw a white ball we flip the fair coin. Now suppose we observe 10 heads. What is the probability we are flipping the unfair coin? It isn't as easy with the Fisherian approach. In this case, we have to factor in the probability of drawing the black ball from the urn. Factoring in this information and using Bayes Theorem it turns out the probability that we are flipping the unfair coin is 0.0003569. The reason is because the draw from the urn means we are almost surely flipping the fair coin.

So what is wrong with the Bayesian approach? Well a bunch of things according to Dembski, but I am only going to focus on two of them to show that either he is being dishonest or he doesn't know what he is talking about. The first is when he writes the following,

When the Bayesian approach tries to adjudicate between chance and design hypotheses, it treats both chance and design hypotheses as having prior probabilities and as conferring probabilities on outcomes and events. Thus, given the chance hypothesis H, the design hypothesis D, and the outcome E, the Bayesian theorist attempts to compare the posterior probabilities of H and D on E (i.e., P(H|E) vs. P(D|E)). If the posterior probability of D on E is greater than that of H on E, then E counts as evidence in favor of D, and the strength of that evidence is proportional to how much greater P(D|E) is than P(H|E). Unfortunately, calculating posterior probabilities requires knowing prior probabilities (i.e., P(H) and P(D)), and often these are not available. In that case, one may merely calculate the likelihoods of E on both H and D (i.e., P(E|H) vs. P(E|D)).

There’s a stripped down version of the Bayesian approach known as the likelihood approach that essentially ignores prior probabilities and simply looks at the likelihood ratio (i.e., P(E|H)/P(E|D)) to determine strength of evidence in favor of a hypothesis. This, however, makes for an idiosyncratic understanding of evidence. Evidence, as usually understood, refers to what causes us to revise our beliefs. But likelihoods ratios are in no position to do that without help from prior probabilities.

This is, in my view, somewhat misleading. Dembski wants the reader to think that there are problems coming up with prior probabilities, and he is right. But what he fails to tell the reader is that the very same source of information for prior probabilities is the source of information for Dembski's notion of specification,

If we can spot an independently given pattern (i.e., specification) in some observed outcome and if possible outcomes matching that pattern are, taken jointly, highly improbable (in other words, the observed outcome exhibits specified complexity), then it’s more plausible that some end-directed agent or process produced the outcome by purposefully conforming it to the pattern than that it simply by chance ended up conforming to the pattern.

In other words, if our background information allows us to select certain outcomes that are specified such as 10 heads or 10 tails. These are specified because our background information with fair coins says this result is strange, unlikely, etc. But this "background information" is precisely where prior probabilities come from. Prior probabilities represent what we think the probabilities of some event occuring prior to seeing any data. For example, we know there are many more fair coins in the world than unfair coins. Hence if we find the coin on the sidewalk we'd be justified in setting a low prior probability for the coin being unfair. And if all else fails we can set the prior probabilities to what are called "non-informative" priors. Interestingly enough, in things like linear regression, non-informative priors and Bayesian analysis yield the "Fisherian" estimates of the parameters. But apparently this is a Bad Thing according to Dembski (yes, in a sense he is arguing against himself).

The other problem is when Dembski looks at the kind of evidence necessary to reject the fair coin/chance hypothesis with the Bayesian approach. Dembski writes,

(5) Backpedaling priors. As a variant of the last point, return to the earlier example of an urn with a million balls, one black and the rest white. As before, imagine that a fair coin is to be tossed if a white ball is randomly sampled from the urn but that a biased coin with probability .9 of landing heads is to be tossed otherwise. This time, however, imagine that the coin is tossed not ten times but ten thousand times and that each time it lands heads. The probability of getting ten thousand heads in a row with the fair coin is approximately 1 in 10³⁰¹⁰ and with the biased coin approximately 1 in 10⁴⁵⁸ (with ten thousand tosses, heads are bound to turn up for either coin). A Bayesian analysis then shows that the probability that a white ball was selected is approximately 1 in 10²⁵⁴⁶ and the probability that the lone black ball was selected is 1 minus that minuscule probability.

Should we therefore, as good Bayesians, conclude that the black ball was indeed selected and that the biased coin was indeed flipped (the selection of the black ball being vastly more probable, given ten thousand heads in a row, than the selection of a white ball)? Clearly this is absurd. The probability of getting ten thousand heads in a row with either coin is vastly improbable, and it doesn’t matter which urn was selected.

First off, what does Dembski mean by "which urn was selected."? There is only one urn in his example. Was he "plagarizing" some of his earlier writing with two urns?

The second problem is far more serious, IMO. These probabilities that Dembski is talking about aren't absurd because the Bayesian approach is absurd, but because Dembski has set up the example so that the prior probability is so close to 1. The closer the prior probability for a given hypothesis is to 1, the more extraordinary the evidence/data is going to have to be to over turn that prior probability. For example, suppose we had 10 balls with 5 black and 5 white in the urn. Now our prior probability is 1/2 for getting the unfair coin. Now what is the probability of having flipped the unfair coin, given ten heads? That is what is

P(Unfair|10 heads) = 0.997.

In other words, we are pretty darned flipping the unfair coin. So by selecting such an extreme prior the data necessary to "swamp" that prior also has to be extreme. But, there is another way of looking at this point. What this says is that even with extreme priors, given enough data the extreme priors can be overcome. This isn't a flaw, it is a feature! So while prior probabilities are important, and selecting an extreme probability can make it very hard for the evidence to rule out the hypothesis, it is still possible for it to happen. The other lesson is to try not to select extreme priors unless you are really justified in doing so. In Dembski's example, the extreme prior is justified due to how he set up the example. But to then turn around and feign shock and dismay over the amount of evidence necessary to overcome the extreme prior Dembski himself set up is like a used car salesman feigning shock and dismay that one of his rust buckets broke down once off the lot.

These two "problems" that Dembski has highlighted are not problems, or they are not fatal problems and/or they are also present in Dembski's own methods. The Bayesian/Likelihood approach is still the best approach. Further, Dembski's approach has other problems he doesn't even consider; such as considering data that might have happened, but didn't. We could have come up with 10 tails and we should consider that too, even though such data didn't come up. What about the rest of Dembski's complaints? In my view they are without much merit, and given his bungling of the issue of the importance of prior probabilities and inability to interpret the results of his own examples correctly this should not be surprising.

Update:Dembski's notion that the 10,000 heads outcome is absurd and neither coin was randomly tossed is of course just silly. Here he sets up an example, then he wants to come up with an outcome that is exteremely unlikely and say, "See, see! That Bayesian approach is no good!" I submit that if we had done the Fisherian test we'd be unlikely to say that the results are any good. Why? Because the results are so out of whack. Let me be clear on this point. Suppose we conduct the test just as Dembski has laid out and we do our Fisherian test with 10,000 heads. We'd reject chance and conclude design. Alternatively we do the experiment and the subsequent Bayesian test and reject chance and conclude design. But, in both cases these conclusions would be unwarranted. Why? Because something went wrong with the test. That is what Dembski is implying when he says such results are absurd. Yes, they are and the results should be jettisoned with either approach. In short, this isn't just a problem for the Bayesians, it is also a problem for those using the "Fisherian" approach as well.

And yes, there is an "escape-hatch" for the Bayesian. The Bayesian analyst didn't condition on all the relevant information. For example, suppose both coins are unfair in that they are two headed. Once we condition on that information, the results are no longer absurd, but are quite reasonable. Are we using a "Fisherian" test here? I guess, but then again there are instances when the Fisherian method does prove useful (such as nonparametrics).

Posted by Steve at November 17, 2005 12:29 AM