P value is a fascinating thing. Everybody loves it, and hates it. When the p value is less than 0.05 (probably out of luck), everybody gets excited; but when it is not, everybody starts mumbling the F word. For many number crunchers, it’s the way of life.

Here is an interesting graph I made in Excel. The x-axis is the z value from a standard normal distribution, and the y-axis is the p value. Based on this figure, one can reach several conclusions:

1) The p value decreases quickly from 1 to 0.10. That says when z value changes a little, the p value decreases a lot; in other words, there is practically not much difference between p=0.9 and p=0.1. They are all non-significant.

2) The curvature from p=0.1 to 0.01 is the turning point. Compared with those change in the previous stage, the z value for p=0.01 is much bigger than the z value for p=0.1. This means that it takes a lot of efforts (big z value difference) to make the result significant (p<0.05). From this point of view, R.A. Fisher is really a genius.

3) The last segment is more interesting. Starting from p=0.01, the curve is flat. Even when the z values change a lot, the absolute change of p values is not big. On the other hand, when we compare p=0.01 with p=0.0001, we tend to think about it in the relative scale. Here we say that the p value has changed 100 folds. But the z value only changes from 2.7 to 4. If one goes down the road, a little change in z value may have more than 100 folds of decrease in p value.

The implications of this exercise are:

1) When the result is not significant, say p >0.10, it really means no difference, especially when the result is from large samples.

2) However, if p<0.05 is significant, the p=0.07 should also be commended and appreciated because a change from 0.07 to 0.05 is very hard. It takes a lot of z values.

3) The 100 folds change of p value from p=0.01 to p=0.0001 is not that significant when considering the absolute change of z values. A change of 100 folds in p values doesn’t mean there are 100 folds of real difference in actually measurements. Even when you can do some multiple test adjustments such as Bonferroni method, it really doesn’t help a lot.

4) When one has a large sample size (say more than 1000), the standard error becomes very small compared with the mean difference. It is easy to get a p value less than 0.0001. But the absolute change in measurements is often very small and of no practical meaning.

5) Here is the darkest thing. Everybody loves small p values. If you can decrease the p value by 10 folds with a little tweaking, your results will be more likely to be accepted by the publisher. It happens that the “fine tuning” is not much difficult if you know how.