Check Your Knowledge
RA Fisher and the P-value
A researcher compared mean pain scores between an intervention and a control group and reported a value of -2.50 for an unpaired t-test statistic (64 degrees of freedom). What is the associated p-value and what conclusions might you draw from this?
Answer: From GraphPad
P Value Results
The two-tailed P value equals 0.0150
This result is unusual if the null hypothesis of no treatment difference in mean pain score is true. By conventional criteria (alpha = 0.05) we would reject the null hypothesis and claim that the mean pain scores differ significantly.
The p-value is often incorrectly interpreted and has many critics. As commented in the Nature article given below,
"P values have always had critics. In their almost nine decades of existence, they have been likened to mosquitoes (annoying and impossible to swat away)"
For more on this check out the following publication:
The Transposed Conditional
One common mistake when with working with p-values is to treat the result as a probability statement about the truth of the null hypothesis. Recall that the p-value is the probability of seeing the observed test statistic or more extreme when the null hypothesis is true. Note that since we have assumed that the null hypothesis is true it is difficult to then use this to make a statement about the probability that it is true! In simple notation we can write this as a conditional probability:
p = P( Sample | H is true)
where Sample is the observed statistic and H is our null hypothesis. Note that what we often want to know is the probability that the null hypothesis is true given our sample statistic. This is
P( H is true | Sample)
Unfortunately, these two statements (with a "transposed conditional") are not the same. As simple example is that if it is raining then there is a high probability that it is cloudy. However, the fact that it is cloudy does not carry the same high probability that it is raining.
P(Cloudy | Raining) is not equal to P(Raining | Cloudy)