Null Hypothesis, p-Value and Significance Level

Answering a question with yes or no

Often, we do not want to estimate a value but to decide: is this drug more effective than the placebo? Is this coin rigged? A hypothesis test organizes this decision rigorously, by confronting the data with a cautious starting hypothesis.

The two hypotheses

• The null hypothesis H0 is the hypothesis that "nothing special is going on": the drug changes nothing, the coin is fair. It is the default viewpoint, the one we try to refute.
• The alternative hypothesis H1 is what we may want to demonstrate: the drug works, the coin is rigged.

We never "prove" H0: we look at whether the data are surprising enough to reject it.

The p-value

The p-value is the probability of observing a result at least as extreme as the one measured, if H0 were true. A very small p-value means: "what I observed would be very unlikely by mere chance under H0" — it is an argument for rejecting H0.

Distribution of the result IF H0 is true (bell curve):

                     _____
                  /       \
   //////|      /           \      |//////
   //////|    /               \    |//////
   //////|__/                   \__|//////
   -------+----------|------------+---------
   rejection    observed      rejection
   region       value?        region
   (left)                     (right)

(the hatched //// regions at the extremes = 5% in total: if the result falls there, we reject H0)

The decision threshold

We set a threshold in advance (often 5%, denoted alpha). The rule is simple: if the p-value is below the threshold, we reject H0 ("significant" result); otherwise, we cannot reject it. Setting the threshold BEFORE looking at the data is essential to avoid cheating.

Classic pitfall

A p-value is NOT the probability that H0 is true, and a "non-significant" result does not prove that H0 is true: it only means we lack evidence to reject it. Absence of evidence is not evidence of absence.