Philip and I were discussing the design of my dissertation experiment, and he pointed me at an interesting book chapter titled “The Null Ritual: What You Always Wanted to Know About Significance Testing but Were Afraid to Ask“. It’s fascinating reading, as it walks through a lot of false beliefs about significance testing as used by psychologists in experiments. I found that my understanding of significance testing was definitely incorrect in the ways described in the chapter.
The “null ritual” from the title is described as:
- Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify
the predictions of your research hypothesis or of any alternative substantive hypotheses.
- Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis.
- Always perform this procedure.
The problem is that the null hypothesis test is p(D|H0), or the probability of obtaining the observed data given that the null hypothesis is true. When doing an experiment, any real world scientist will have a hypothesis that they are testing and usually hope that they can prove that it is true using the data from the experiment. What we really want is p(H1|D), or the probability of our hypothesis being true given the observed data. However, we need Bayes’ rule to draw a conclusion about the hypothesis and that requires the prior probabilities of the hypotheses, which are often not available to us beforehand.
The chapter also brings out the controversies in statistics between different approaches and goals of particular techniques, which is usually glossed over in teaching of statistics.
I’m planning to follow the authors’ recommendation in my research: “In many (if not most) cases, descriptive statistics and exploratory data analysis are all one needs.”