I agree with Richard on this. I think that going fully Bayesian is the better way.

My use of Gigerenzer’s “Natural Frequencies” point of view is pedagogical. It is an easy way to introduce the ideas I want to present. One example is one that I often use when people ask me what I do and I want to give them an example of Bayesian stats (many will have had standard stats courses, which they hated)…I give them an example.

I tell them (correctly) that in 90% of the cases where a woman (in the general population) has breast cancer, a mammogram will come up positive, and in 90% of the cases where she does not have breast cancer, it will come up negative. I then ask, what if a woman has a mammogram and it comes up positive? How worried should she be?

As Gigerenzer points out, a large fraction of physicians will incorrectly say “90% probability she has cancer”, which is of course wrong. To answer correctly, you need the additional information that in this population about 1% of women have undetected breast cancer (the prior). Then I say, in a group of 1000 women, 10 will have cancer, and 9 of those cancers will be detected by the mammogram. Of the 990 women who do not have cancer, 99 false positives will occur. So in that group, there will be 99+9=108 positives, of which only 9 are actual cancers. So the proportion of women that test positive and have cancer is 9/108, or about 8%. The beauty of this approach is that I can do it in the parking lot, without even pencil and paper, and it will be understood.

Once the students understand this idea thoroughly, it is easy to transition to fully Bayesian calculations.

P.S. For those interested, “Bayesians can regard probability even if the event is unique” also describes hypothetical frequentism. Which begins the debate over what the better way is to do hypothetical frequentism: by just adapting traditional frequentism to it, or going Bayesian on it. Of course, I think the latter.

http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Also, I recommend following Andrew Gelman’s blog, it is one of the few things that I read faithfully. Andrew talks extensively about how things like p-values are misused. Type “garden of forking paths” into his search engine for example.

http://andrewgelman.com

The one thing that is different from frequentism is that Bayesians can regard probability even if the event is unique, i.e., not the result of something repeated over and over. For example, a given football game is played only once. It still makes sense (if you are into this sort of thing) to estimate probabilities of a given team winning this game and thus to make bets at certain odds (but not other odds) as rational bets.

…what is done in practice in data analysis.

Which is fine. I just don’t want anyone to mistake that for being the same thing as modeling and testing historical inferences.

You are describing a method correctly, that doesn’t pertain to ordinary inference procedures.

This is rather like teaching someone calculus, when they ask you how to do geometry. Or, like I said, advising someone commute to work on a three stage rocket, when walking will do.

Most inferences in philosophy and history (and indeed even sometimes in the sciences) do not benefit from extremely advanced and elaborate procedures. Yes, those procedures are useful, when you have the data that they are designed to get the best out of. But they are far too complicated for most purposes.

No, that I don’t know the exact reference for.

In a sense, you are right: Frequentism and Bayesianism are not so at odds as the battle-lines claim. But there are significant differences, which seriously matter (to life and limb even, as changes made in medical science and research due to introducing Bayesian reading have recently shown; but also to determining the actual rate of false positives in science research, which is being discussed everywhere in the literature now, as people realize the limited use of p-value reasoning). And there are general lessons one learns from thinking like a Bayesian, as it represents a complete logic, the entire machinery of a scientist’s inference from premise to conclusion, whereas frequentism by itself conceals much of that apparatus, leaving scientists prone to self-deception and error.

I’m not clear on how Marilyn Vos Savant on the Monty Hall problem comes into it. But this is disconcerting because I could never understand any of her explanations. Nor could I ever understand how Bayesian reasoning does anything other than confuse the issues. For me, it was forgetting all the stuff about new information and updating that finally made it clear. Assuming random distribution of prizes, given enough trials, the frequency of any given door (left, right or center) hiding the grand prize would be 1/3. This real frequency does not change. The real frequency with which the grand prize would be found behind another door then is 2/3. That doesn’t change either. When Monty Hall opens one door, and offers a chance to switch to another door, the frequency with which the prize is found behind another door is still 2/3. Of course you switch. You can’t just update the information to think, two doors, therefore a frequency of 1/2 in a large series of random trials. Evidently what Bayesians means by updating is not so evident to some of us. Are we still Bayesians?

Prompted by this post I bought a copy of Jordan Ellenberg’s recent book and re-read the chapter on Bayesian inference. When Ellenberg declared that Bayesian reasoning leads you to take Nick Bostrom’s notions of simulated universes seriously, it inspires grave reservations. (This is similar to the recent declaration that Bayesian reasoning indicated some high probability the multiverse was real!) Bayes’ theorem is so easily read as frequentist, that it is tempting to read it as a necessary corrective to mechanical use of null hypothesis and p-values. But as someone observed (Mario Bunge I think,) observed, propositions don’t have probabilities. Assigning prior probabilities to the data set of evidence is the hard work of reasoning but it’s not clear how Bayes’ theorem helps this.

Bill asked how a person would typically perform inference when there was uncertainty about the probabilities, and I simply tried to give the answer suggested by Bayes theorem and what is done in practice in data analysis.

Re. “this is exactly what I recommend in PH (and run in parallel to the odds method in OHJ). The concern that our min/max is going to be too wide is a category fallacy. It is simply the case that in history, our margins of error often are that wide.”

As I said, I don’t want to comment on what historians ought to do or ought not to do, however in Bayesian data analysis you wish to see how your modelling assumptions (such as the uncertainty in the probabilities) translate into uncertainty in the quantities of interest. The way to do this is using Bayes theorem, and it will lead to different answers than the min/max approach — answers which from a Bayesian perspective is are spurious. You can say that since we are doing history we ought to do something else, however I was simply trying to provide the standard answer. If you are interested I can recommend guides for how parameter estimation is normally done using Bayes theorem?

Re. “I don’t fathom what’s ad hoc about calculating the a fortiori posterior and calculating the a judicatiori posterior. The full theoretical foundation for this comes from the philosophy of history and is detailed in PH. Quite simply, this is necessarily what you must do when the two questions you are asking are, “what is the highest probability of being true that I can reasonably believe this claim has?” and “what is the lowest probability of being true that I can reasonably believe this claim has?”
–
Well, by ad hoc I simply mean a method for manipulating probabilities that is not Bayesian. From a data-analysis perspective, you wish to know how probable different values of the quantities of interest are under the assumptions you have made (model assumptions, assumptions about uncertainty in probabilities, whatever) and Bayes theorem gives you a way to compute this. If you rather do the min/max procedure, you get numbers that do not represent this and are not computed using Bayes theorem. You can say this is what you want in history, but it is certainly not what you want in data analysis.

Cheers,
Tim.

I found the article you cited containing a comment by Monty. I was looking at the wrong link. Here it is:

http://priceonomics.com/the-time-everyone-corrected-the-worlds-smartest/

I still can’t find the letter in American Statistican! If I do find it I’ll let you know.

Thanks, Richard!