Another attack on frequentist statistics
A news report in nature tells of yet another study concluding that Bayesian statistics are better than frequentist statistics. **Disclaimer: I don’t have time to read the actual scientific paper being reported, so the opinions that follow are about the Nature news report, not the original article**
John Ioannidis wrote a great paper a few years ago called “Why most published research findings are false”. However, Nature quotes his response to this new article, which to me is just too simple minded. Sure it could have been taken out of context, but in any case it is not a message I support,
“The family of Bayesian methods has been well developed over many decades now, but somehow we are stuck to using frequentist approaches,” says physician John Ioannidis of Stanford University in California, who studies the causes of non-reproducibility. “I hope this paper has better luck in changing the world.”
I will repeat my opinion on this kind of thing: (1) frequentist statistics are neither perfect nor terrible, (2) Bayesian statistics are neither perfect nor terrible, (3) it is possible to cheat with Bayesian statistics, (4) it is possible to cheat with frequentist statistics, and in conclusion, (5) the problem is not with this or that particular statistical paradigm, but rather with researchers really wanting to find results that are interesting…and therefore making interesting conclusions however they can (whether they is any truth to those conclusions or not).
Blaming a particular statistical paradigm is just a red herring. If we want science to be more reproducible, the scientific reward system needs to shift in favour of skepticism. This will have its downsides too, because if we don’t reward scientists making bold claims then science could be boring and may in fact fail to notice subtle but in-the-end interesting results. Of course the price of rewarding scientific boldness is many published results that are untrue.
The problem of how to encourage better scientific practice is at the intersection of the sociology of science and statistics (and methodology more generally). If you ignore one of these pieces (e.g. this recent Nature news report coming down on an entire statistical paradigm), then you will necessarily be oversimplifying the problem.
ecology & stats 
Steve, what is your opinion about the relative merits of Bayesian vs frequentist statistical approaches? I really wish you would weigh in on that.
🙂
In general it depends on the application, but I think we need both approaches. What I think is definitely wrong, as I’ve tried to point out in this post, are wholesale rejections of one side in favour of the other. The anti-Bayes people and the anti-frequentist people are just ignoring large amounts of evidence that both ways of thinking have contributed to the correction of bad statistical reasoning. It seems like these extreme positions are tantamount to claiming that it isn’t raining while one is outside being rained on (to paraphrase Rick Mercer’s rant about our Prime Minister last night — you may need to be Canadian to understand this reference).
Its like this…Bayesian statistics allows us to make more conservative estimates, by allowing us to put conservative priors that aren’t fooled by noisy and overly extreme maximum likelihood estimates (or overly extreme unbiased estimates). Bayesians are allowed to say things like ‘I just don’t believe that this data set is representative of the entire population, because the estimated effect without the prior is just unbelievably large…but…maybe with more data…we’ll see…’ This I think is an under-appreciated point, especially by Bayes-haters. Prior distributions almost always make us more conservative, not less. Sure we can cook up priors that overwhelm the data and give us the answers we want, but no reputable Bayesian does that.
However, the whole subjective Bayesian philosophy movement was, although very interesting and useful, in the end a research programme in the wrong direction (but now we know so it was actually quite useful!). Bayes theorem does not solve all of our problems. As Bradley Efron said “using Bayes theorem isn’t the problem, its always using it”, or something like that anyways. On the other hand, frequentist statistics are very useful for identifying when a model (null or otherwise) is just not a plausible description of the statistical population of interest. Frequentist stats in other words are great at model checking, which Bayesian stats can be really bad at sometimes. One of my first statistical mistakes/lessons was thinking that a posterior distribution is all you need to make good inferences. This is true, as the subjective Bayesians proved, but only insofar as the prior-likelihood combination provides a good description of the target statistical population (e.g. it cross-validates well; generates data consistent with observed data).
Here’s a synopsis of my opinion:
https://stevencarlislewalker.wordpress.com/2012/10/23/basis-functions-for-locating-statistical-philosophies/
Great answer Steve!
For the record, I wasn’t being serious; I know that you favor a context-specific approach, one that requires actual thought, not formulaic application, one which to me makes great sense. But you added some additional points here that, to me, are really helpful, in particular that Bayesians can say something about the probability that one’s ML-based estimates are actually drawn from the real target population of interest. That is exactly an issue I’ve had on my mind, which I think applies to statistical analysis generally, i.e.,are you *really* testing the same system that you, or somebody else, did before? And how can you be sure of that? But then, what do I know about it.
My sense of humour is not what it used to be 8) I guess I was just moving too fast to register the smiley face.
‘are you *really* testing the same system that you, or somebody else, did before?’
No. At least not if you are an ecologist.
“No. At least not if you are an ecologist.”
Which is more than a bit of a wrench in the conceptual machine.
Bravo! I think you have done a superb job here.
In my own branch of chemistry we don’t generally use statistics, even where might benefit from its application, and we certainly don’t waste time teaching statistics to our students. That said, I have found that the bayesian probability calculus provides a useful conceptual tool for thinking about my work.
And we certainly have the issue that bold hypotheses get published, even if you have to lie about it in your submissions, while boring-yet-correct hypothesis gather dust in the basement. And to make matters worse, a lot of idiots in positions of authority have browsed a copy of Kuhn and decided that the sole purpose of a research program is to initiate the next scientific revolution.
And reproducibility is an odd beast. I have two favorite examples. In the invention of the Ziegler Natta catalyst, a lump of polyethylene was made and its existence taunted the researchers for months while they figured out how to do it again. In a proprietary case, one of my colleagues stumbled over a reaction which led to the development of a whole new class of chemical transformations – but no one was ever able to replicate the original founding reaction.
Which is to say, regarding the initial issue of repeatability, there are categories of science in which we are confident that something is true because of the preponderance of supporting evidence even though we can not directly replicate the original experiment.