My Problems with p-values

This post is the fourth in a series of six posts in which I am arguing against the use of p-values for reporting the results of statistical analysis. You can find a summary of my argument and links to the other posts in the first post of the series. In this post, I present my problems with p-values. 

What are my problems with p-values? Oyy, where to start? Here are four reasons why I hate p-values and NHST:

1. Science is not about making decisions each time you see a new sample. NHST and p-values are not designed for scientific inquiry but for industrial decisions.
2. In general, scientists interpret statistically significant effects as equal to the true effect while non-statistically significant results are interpreted as zeroes. Both of these interpretations are deeply WRONG.
3. Statistically significant results are biased.
4. Marginally significant results are very imprecise.

NHST and p-values are not designed for scientific inquiry but for industrial decisions

NHST and p-values are not adapted to scientific inquiry but to industrial practice. In order to explain to you why this is the case, let me tell you a story.  (I do not know if this story is 100% true or only apocryphal, but I'm pretty sure I read it in one of Stephen Stigler's books or in Hald's History of Mathematical Statistics. Whether the story is accurately true or not does not really matter though, since it is mostly there to illustrate the context of use of NHST and p-values).

The inventor of the significance test is William S. Gosset, better know under his pen-name of Student (he was so humble that he considered himself a student of the great statisticians of his day, especially Karl Pearson). William Gosset wrote under a pen-name because he was not an academic but was working for a private firm, actually the famous Guiness beer factory. Gosset designed the testing procedure in order to solve a very practical problem that he faced at his job. Everyday, a new batch of grain would come in. Before sending the grain into production, Guiness employees would take a sample of the grain (let's say ten small samples taken in random parts of the batch) in order to assess its quality. They would estimate the quality of the grain in each of the samples for a characteristic important for brewing. Based on the sample, they would have to decide whether to discard the batch or put it under production. The problem is that the sample is a noisy estimate of the quality of the batch. If the batch was bad, but they wrongly decided to put it into production, they would lose money. If the batch was good and they decided to discard it, they would also lose money. You recognize the errors of the second and first type of test statistics. So Gosset had to make a choice, everyday, based on a sample, to discard or accept a batch of grain. He devised a procedure that would minimize the risk of discarding a good batch under a fixed probability of discarding a bad one. The procedure simply used a test statistic: compute the value of the test statistic under the assumption that the batch is good. Compute its p-value. Discard the batch if the p-value is smaller than 0.05.

Gosset's procedure (and test statistics in general) makes a lot of sense in an industrial context. There is repeated sampling and actual decisions made at each sample repetition. Test statistics are perfectly adapted to this problem. Science is a very different problem altogether. There is no repeated sampling. We do not take a decision after each sample repetition. We do not need a procedure to help us make this decision. Fisher was the one who adapted Gosset's idea and translated it to scientific practice. He devised p-values as a means to estimate the strength of the evidence in favor of an assumption. He suggested that we could say that under 5%, the bulk of the evidence could be considered overwhelmingly against the assumption. But he never made this threshold a magical threshold. What made this threshold magical was the procedure of decision attached to statistical testing, that Neyman and Pearson coined after Gosset. But, again, this procedure was adapted to an industrial context of repeated decisions, not to scientific inquiry.

NHST and p-values give a false cutoff sense of confidence

The problem with using test statistics in science is that they focus our attention on the position of our results with respect to a cutoff. Have you ever noticed how much more excited you feel when your results cross the 5% significance threshold? How disappointed you are when they are just below? We also tend to radically alter our reporting of a result when it is statistically significant. For example, if  a coefficient is statistically different from zero, we are happy and we report it as being a positive effect. If the result is not statistically different from zero, we report as being insignificant, and in general we consider it as good as zero. This is something every single one of us has felt.

And this is wrong. It proves a deep misunderstanding of what sampling noise really is and what statistical testing is all about.

Look for example at samples 27 and 28 when N=1000 on the figure above. With sample 27, you have a treatment effect estimated at around 0.18, significantly different from zero at the 5% level. So two stars significance. Great. You tend to interpret this result as a 0.18 positive significant treatment effect, and you are going to remember 0.18. With sample 28, you have a a treatment effect estimated at around 0.17, not significantly different from zero at the 5% level. You tend to interpret this result a non-significant treatment effect, and in general you are going to remember it as a zero. But the two samples contain exactly the same objective information: the confidence interval for the effect is large, ranging from very small (zero or slightly negative) to large.

You cannot change your opinion on a program because some random noise has marginally changed your estimator so that its test statistic falls just above or just below 1.96. Nothing has objectively changed between these two samples. The only reason why we would need to choose a cutoff and change our minds when crossing this threshold is because we want to make a decision. But there is no decision to make. So we should consider the two samples as bringing exactly the same information: either a very small effect (positive or negative) or a very large positive one.

But the 95% confidence interval for sample 28 tells us that the effect might be negative whereas that is not the case for sample 27. Doesn't it count for something? No, and for two reasons. First, a very small effect, positive or negative, is just small and does not have important consequences in the real world. Second, even if it does, your precision does not allow you to conclude anything certain. Probability distributions are continuous here, and the change of probability for the treatment effect being below zero from sample 27 to sample 28 is marginal, extremely small. You can see that if you use the 99% confidence interval. Then sample 27 also contains zero and small negative effects.

Look now at samples 18 and 19 of the same graph. The effects there are small and zero is well within the confidence bands, so a statistical test would just give you an insignificant estimate. In general, you will interpret this as a zero. But this would be wrong. Completely wrong actually, since the true treatment effect is actually 0.18. And the objective information from the confidence interval tells you just that: 0.18 is well within the confidence bands too.

Stick with the objective information. Tests focus your attention on details and marginal changes and cutoff decisions instead of looking at sampling noise objectively. Tests are used as a way to gain false certainty in front of sampling noise. No statistical test can get rid of sampling noise.
 

Statistically significant treatment effects are biased

One very annoying property of statistically significant results is that they are always biased upwards, especially if sampling noise is large (and thus especially if sample size is small). Look again at the figure above. With N=100, the estimates that are statistically different from zero at the 5% level are 2 to 2.5 bigger than the true effect. With N=1000, not all statistically significant results overestimate the true effect but most do.

That statistically significant results are biased upwards is a mechanical consequence of NHST. In order to shed more light on this, let's compute the p-values for the two-sided t-test that the treatment effect is zero for all our Monte Carlo samples using the CLT-based estimates of sampling noise. The figure below presents the results.

You can see that with N=100, treatment effects are significant at the 5% level only when they are at least bigger than 0.3 (the 5% threshold is the blue line on the graph, the red line is the threshold for 1% significance). Remember that the true effect is 0.18! With N=1000, samples with an estimated effect smaller than 0.1 are never significant at the 5% level. As a consequence, if you take the average of the statistically significant effects overestimate the true effect by a large amount: with N=100, statistically significant effects are on average double than the truth, whereas with N=1000, they are roughly 50% bigger.

Note that there is no such problem for larger sample sizes, where all results are statistically significantly different from zero. Actually, we should expect some estimates to be close to zero, but the probability that it happens is so small that it has no practical effect. People accustomed to p-values are thrown off when using large sample sizes where everything is significant at conventional levels. This is actually a funny consequence of people not understanding sampling noise and test statistics: the fact that everything is significant means that uncertainty about parameter values has decreased and that you can actually look at the magnitudes of the coefficients, not whether they are different from zero.

Marginally significant results are very imprecise

Another related very unfortunate consequence of p-values is that results that are marginally significant at the 5% level have are very imprecise: their signal to noise ratio is equal to 0.5, meaning that there twice as much noise as there is signal. This is a very simple consequence of using NHST. Remember that scientists consider a result to be significant at 5% when the ratio t=|x|/se(x) is superior to 1.96. Now, signal to noise ratio can be defined as s/n=|x|/y, with y the width of the 95% confidence interval. Remember that with a normal distribution, we have se(x)=y/(2*1.96). As a consequence, t~1.96 -> s/n~0.5.

A related consequence of using NHST is that, when choosing sample size using a power analysis based on a similar testing procedure, the signal to noise ratio using default settings for size (5%) and power (80%) is also small. Indeed, one can show that the signal to noise ratio of a power analysis for a one sided test (for a two-sided, replace alpha by alpha/2) is equal to:

with delta the confidence level used to build an estimate of sampling noise. With delta=95%, and a two-sided test, the signal to noise ratio of a usual power analysis is 0.71, so that there is still 1.4 times more noise than signal.

How p-values Work in Practice

This post is the third in a series of six posts in which I advocate against the use of p-values to report the results of statistical analysis. You can find a summary of my argument and a link to all the posts in the series in the first post of the series.

In the previous post of the series, I explained what is sampling noise. In this post, I am going to explain how p-values are used to report the statistical significance of results. I'll describe the practice in economics, but I believe most fields use them in a similar way, especially within the social sciences.

Researchers generally check for statistical significance in the following way: they divide their estimated effect x by its standard error se(x) (in general, se(x) is equal to y/(2*1.96), where y is the width of the 95% confidence interval, the estimate of sampling noise I used in my previous post), take the absolute value and compare the resulting ratio to the following thresholds:

• If the ratio is above 2.57, the p-value is below 1%, we say the effect is significant at 1% and some statistical software puts three stars in front of the estimate.
• If the ratio is above 1.96, the p-value is below 5%, we say the results is significant at 5% and some statistical software put two stars in front of the estimate.
• If the ratio is above 1.68, the p-value is below 10%, we say that the effect is significant at 10%  and some statistical software puts one star in front of the estimate. 

The rationale behind this approach is based on Null Hypothesis Significance Testing (NHST). Before seeing the data, you commit to reject the Null assumption of zero effect if and only if the probability that the true effect comes from a distribution centered at zero is low enough. The procedure that I describe above is called a two-sided procedure in which you do not want to commit ex ante to a privileged direction for the effect of your treatment. It can be either positive or negative and you want to be able to detect either of these effects, and you allocate the same power to both.

The graph below illustrates this procedure with an example taken from my class. The true distribution of the effect across random samples is in black (and nicely approximated by a normal distribution in blue) and is centered around the true value of the effect, in red (0.18). The problem of sampling noise is that you do not know where the true distribution is centered, and you only have access to one point from the black distribution. Thanks to the central limit theorem, you can estimate the width of the black distribution and you can approximate it by a normal, but you still do not know where to center it. Here comes NHST. With NHST, you assume that the true distribution is centered at zero (the green distribution in my graph), and you compute the threshold t for which the probability of falling above t or below -t is equal to say 0.05 when the distribution is centered at zero. This gives you the two green discontinuous lines around 0.1 and -0.1. The idea of NHST is to say that if the estimated parameter falls above t or below -t, it is very unlikely that it comes from the green distribution (over sampling replications, it will only happen 5% of the time if the green distribution is true).

Based on the notation I used in the previous post of the series, with x the effect of the treatment and y the (95%) sampling noise, you can compute the p-value associated with the two-sided test as follows: 2*(1-Phi(|x|/(y/(2*1.96)))), with Phi the cumulative distribution of the standard centered normal. With the example I used in my introductory post, we have x=0.1, y/2=0.09996, so that the standard error is equal to 0.051 and the p-value computed is 0.0498. So the result is said to be statistically significant at 5%.

What is Sampling Noise?

This post is the second in a series of six posts in which I am arguing against the use of p-values for reporting the results of statistical analysis. You can find a summary of my argument and links to the other posts in the first post of the series. In this post, I introduce the problem of sampling noise, the issue that p-values are trying to solve. It is only when I started to understand better what sampling noise really is that I started to hate p-values so much.

What is sampling noise? Let’s take the example of a Randomized Controlled Trial (RCT). In a RCT, we randomly allocate individuals to two groups: one that receives the treatment of interest (drug, job training program, ...) and one that does not. We interpret the difference in average outcomes between the treatment and control groups as the causal effect of the treatment. The common intuition behind a RCT is that treatment and control groups are identical in every respect except as to whether they receive the treatment. But actually, when you run a RCT, the treatment and control groups are identical only when their sizes are infinite. In real life applications, with finite sample sizes, treatment and control groups differ. Some confounding variables are distributed differently in the treatment and control samples and they bias the estimator of the treatment effect. Thanks to randomization, there is no systematic direction to this bias, and it is null on average over sample replications, a property that we call unbiasedness. But in a given sample, the very sample that you might have inherited and that you might be using, the size and direction of the bias are unknown. Knowing that it is zero on average is a poor consolation.  You’re not dealing in averages, you’re dealing with the sample that you have.
 
Here is an illustration from my class. In order to build this illustration, I generated 1000 random allocations to a treatment and control group for four different samples of increasing size taken from the same population (i.e. governed by the same model). For each random allocation, I computed the difference in average outcomes between treatment and controls. The histogram presents the distribution of these estimates. In red is the true effect. (The histogram actually presents the results of drawing a different sample at each replication on top of a different treatment allocation. Both graphs are extremely similar, I just happen to have this one readily available in a nice suitable format. The comparison with the graph obtained with keeping the same sample can be found in Lecture 0 of my course).

With a small sample size (N=100), sampling noise is large and estimates stemming from a given random allocation are extremely imprecise. To the point that almost a 1/4 of the estimates have the wrong sign. In my class, I formally define sampling noise y as the width of the 99% (or 95%) symmetric confidence interval around the true value. You can also use the standard deviation of this distribution as an estimate of sampling noise. Actually, with normal distributions, sampling noise = 5 (or 4 for the 95%CI) times the standard deviation. Here are the estimates of sampling noise for the examples above:

As you can see, sampling noise is large with small sample size and decreases as sample size increases (actually, it decreases with the square root of sample size). With a small sample size, sampling noise is large, precision is low, and a lot of values of the estimated effect might be due to sampling noise. With a small sample, we are not going to be able to rule out a lot of true values of the effect because noise is going to affect our estimates too much. With a very large sample size, noise is trivial and the order of magnitude of the true effect is much more clearly estimated.

So the question then becomes: what can we do when there is sampling noise? At least two things:

1. Make sampling noise as small as possible (the best approach)
2. Quantify the size of sampling noise (when you cannot do 1.).

Let me talk about quantifying sampling noise, because this is what p-values are about. Several of the most important tools in statistics enable you to compute an estimate of sampling noise using information from only one sample. Think about how beautiful this is: you can recover an estimate of an unobserved quantity defined over replications of samples from one unique sample! We have several ways to do that: 

• Chebyshev’s inequality, that gives you an upper bound on sampling noise.
• The Central Limit Theorem (CLT), that approximates the sampling noise of an average by a normal distribution. When combined with other tools like Slutsky’s Theorem and the Delta Method, the CLT enables to approximate sampling noise for estimators that are combinations of sample averages.
• Resampling methods, that use the current sample as a population and mimic the sampling process from this pseudo population. There is the bootstrap, randomization inference, subsampling...

Here is an example from my class where you can see the true sampling noise (in red) along with its estimate from one sample obtained here using the CLT (in blue).

Pretty impressive, right?

Now that we understand what sampling noise is about, and how to estimate it, we will see in the next installment of the series, how p-values deal with sampling noise.

Why I Hate P-Values, Why You Should Too, and What to Do Instead

In my previous blog post on the empirical revolution in economics, when I discussed publication bias, I suggested to ditch p-values. It's a little bit of an extreme position, since we social scientists (and many others) rely almost exclusively on p-values and null hypothesis significance testing (NHST) in order to assess the importance of our results. But after spending a lot of time trying to understand the problem that p-values and NHST are trying to solve, especially in order to teach them in my Econometrics of Program Evaluation class, but also in order to make sense of what I was doing when reporting the results of my own research, I've come to the conclusion that not only p-values and NHST are not useful for solving the problem that we have (namely the problem of assessing sampling noise), but that they are the worse tools that we have to solve this problem, tools that have so many nefarious consequences that I'm now convinced that the best course of action for us social scientists (and others) is to stop using them altogether. 

In order to make my case and present my argument, I am going to need six more posts. In the first one, I present the problem of sampling noise as I understand it. In the second post of the series, I present how p-values solve the problem of sampling noise. In the third post, I present my problems with p-values. In the fourth post, I detail the problems that p-values raise for science, with a focus on publication bias and precision. In the fifth post, I'll present my proposal. My proposal is NOT a perfect solution, and I'm pretty sure there are other approaches that can work even better. In the sixth post, to be written after the complete series is released, I'll compile the remarks I received on my proposal and I'll list the alternative solutions that people have brought to my attention.

If you do not have the patience to read through the six blog posts, here is my case in a nutshell:

1. p-values are bad because they react discretely to sampling noise, instead of accurately reporting it as a continuous measure. As a consequence, people misinterpret the true precision of their results. When a result is statistically significant, they tend to interpret it as saying that the truth is exactly equal to their estimate. When a result is not statistically significant, researchers tend to interpret it as evidence of no effect. Both of these interpretations are deeply wrong. They end up generating bad behavior, with people focusing exclusively on statistically significant results, leading to publication bias and overestimation of effects.
2. I propose to report the estimated effect as "x±y/2," with x the main estimate and y the width of the 95 (or 99) confidence interval around it, and to interpret the plausible magnitudes of the effect that our experiment rules out. This gives the same importance to sampling noise as it gives to the main estimate and focuses on a continuous interpretation of sampling noise. Compare for example these two ways of reporting the same barely statistically significant result: 0.1 (p-value=0.049), 0.1±0.099. The first result gives you the impression of a positive result equal to 0.1. The second one gives you the impression of a very imprecise result, that could very well be zero or double the one you've just found. It could even be slightly negative (remember we are dealing in probabilities). What's sure is that the true effect is probably not very negative et probably not much bigger than 0.2. In between, everything is possible.
3. In order to refine further your analysis, I would recommend to express the magnitudes of possible effects in units of σ, the standard deviation of your outcome (i.e. Cohen's d). What's nice with Cohen's d is that it enables to encode orders of magnitudes of effects: a large effect is of the order of 0.8σ, a medium effect of the order of 0.5σ, a small effect of the order of 0.2σ and a very small effect of the order of 0.02σ. The vocabulary about magnitudes of effects being normalized, you can interpret your results as excluding some magnitudes. For example, with a standard deviation σ of 1, the previous result tells you that there is probably not a large, medium or even small negative effect of your treatment, nor a large or medium positive effect, but that your results are compatible with a very small negative effect, and a very small or small positive effect. Whichever one is true in reality your experiment is not precise enough to tell you. This is not nothing: you have learned something valuable from this experiment. But you have also acknowledged its limitations and lack of precision.