Why lack of statistical significance shouldn’t depress us
Andrew Gelman writes:
With a small sample size, not every comparison is going to be statistically significant.
I know this is kind of obvious, but I feel like ecologists (especially as reviewers!) need to keep this little piece of wisdom in mind more often. In ecology its often very hard work to build up even a modest sample size. During a PhD, two grueling field seasons often generate the kind of sample size that the Higgs Boson people can get in a small fraction of a picosecond. And since causality in ecological systems is a complex web of interactions, we’re usually only looking for subtle effects. Small effects and small sample sizes mean that we should get a whole lot of false negatives, independent of the level of talent of the student.
I think we should all try to remember these basic facts of the lives of ecologists more often when evaluating manuscripts and CVs. A well-written manuscript is a well-written manuscript, regardless of how
interesting significant the results are.