Objectives

Here we take the Which bag? game a stage further, looking at Bayes' Rule and probability. Check the web link to see how we used it to assess the strength of evidence provided by specific likelihood ratios and differences in AIC.

The experiment

Again we have identical small bags, some containing all-white Go stones, others 9 white and 1 black. You take a bag a random.

The aim is to decide whether the bag you took contains all white stones or mixed stones by pulling out one stone at random.

Open the spreadsheet you used to calculate likelihoods and AIC last time (or download this spreadsheet); we'll enter the new data into this.

What flavour of 'probability'?

Last time we did this, we considered the strength of evidence for the all-white hypothesis provided by drawing 1, 2, 3, etc white stones. (If we drew a black stone, we could be certain that there was at least 1 black stone in the bag.) We used likelihoods, the likelihood ratio, and AIC (which is based on log likelihoods) to quantify the evidence, and saw what values correspond to moderate, strong and convincing evidence. We did not talk about the probability that the bag has all white stones.

From the frequentist point of view, the probability that your bag contains all white stones is nonsense; it has all white stones or it does not have all white stones, there's no probability involved. You cannot take that particular bag many, many times and see what proportion of times it has all white stones.

Only Bayesian probability makes sense here: how sure are we that the bag contains all white stones. But Bayesian probability depends on prior information - or prior assumptions.

(For more on the different kinds of probability click here.)

Applying Bayes' Rule

The basic rule is: posterior probability prior probability x likelihood.

To convert our likelihoods into probabilities, we need priors. What is our belief about the probabilities for all-white and mixed before we draw the first stone?

An informative prior

If you knew that 30% of the bags had all white stones and 70% had one black stone, you would base your prior on that. Before drawing the first stone, you'd put Prob(all-white) = 0.3 and Prob(one-black) = 0.7. (By the way, what interpretation of probability lies behind that decision?)

How do these probabilities change as we draw stones?

We start off as before by calculating the likelihoods for each model given the data. These are in rows 4 and 5 in the table below, which records 4 consecutive white stones being drawn, followed by a black stone.

I've added the prior probabilities in cells B7 and B8. These don't change, so we don't need to copy them across to all the columns.

Next, we calculate prior x likelihood for each model: this is shown in the last two rows.

Hint: In cell B10 I typed  = $B$7 * B4  then copied the formula to the right; "B4" changes to "C4", "D4", etc, but $B$7 doesn't change (search Excel help for "Absolute reference" for more information).

To get the posterior probabilities, we need to scale the (prior x likelihood) values so they add up to 1. We do this by dividing each value by the sum of the two:

The "sum" row (row 12) is just the sum of the two rows above it. Then in B14 we have  = B10/B12  and in B15  = B11/B12 .

As we draw out more white stones, the posterior probability of the all-white model increases steadily, and that of the mixed model decreases. After 4 consecutive white stones, the probability has gone up from 0.3 (our prior estimate) to 0.4. After 8 white stones, the posterior probabilities are almost equal; after 30 white stones, you're 90% certain that you have an all-white bag. And of course, if we pull out a black stone, the mixed model becomes a certainty (posterior probability = 1).

That's it! We don't have to worry about what constitutes moderate / strong / convincing evidence; we have the probability that each hypothesis is true.

Without prior information

Suppose you didn't know the proportion of all-white bags in the total population of bags. You still need prior probabilities before you can use Bayes' Rule and calculate a posterior probability.

The usual solution is to allocate equal probabilities to the two models, ie. Pr(all-white) = Pr(mixed) = 0.5. This is sometimes called an "uninformative" or "uniform" prior. Put these values for the prior probability into the spreadsheet instead of 0.3 and 0.7 and see what happens.

Now, with 8 white stones, the posterior probability for the all-white model becomes 0.7 instead of 50-50, and only 20 white stones are needed before you are certain that it's an all-white bag. If you have lots of data (say, 50 draws) the prior might not matter very much (with 50 draws the posterior probabilities are 0.988 vs 0.995), but with sparse data, it makes a big difference.

Comparison with likelihood ratios

This one comes under the heading of "food for thought"!

With the uniform prior we used above, calculate the posterior odds that the bag has all white stones: this is the posterior probability for all-white divided by the posterior probability for mixed.

Compare these posterior odds with the likelihood ratio we calculated in the previous exercise. What do you notice?

Advocates of likelihood ratios vs Bayesian posteriors will tell you that their methods avoid the problem of choosing priors. Maybe what they are really doing is Bayesian analysis with uniform priors, and they're just sweeping under the carpet the question of whether uniform priors are appropriate.

Pros and cons of the Bayesian approach

• If you have sparse data and no basis for an informative prior, you can still use likelihood ratios and AIC to assess the evidence provided by the data.
   
• With sparse data and a good basis for an informative prior, the Bayesian approach will give you a good posterior, incorporating the prior information and the information from your data.
   
• If you have lots of good data, the prior is not very important, and you can use the Bayesian approach with an uninformative prior. You should nevertheless explain why you chose the prior you did, and try different priors to see how sensitive the result is to the choice of prior.

What next?

TBA!