The logic of hypothesis testing
A hypothesis test asks whether data provide convincing evidence against a default claim. It works like a courtroom: assume innocence, then look for strong evidence.
Null and alternative hypotheses
The null hypothesis \(H_0\) is the default claim of no effect or no difference, such as \(\mu = 100\). The alternative \(H_a\) is what we seek evidence for, such as \(\mu > 100\) (one-sided) or \(\mu \ne 100\) (two-sided). Hypotheses are always about population parameters, never about sample statistics.
H_0: \mu = \mu_0 \quad H_a: \mu \neq \mu_0The test statistic
A test statistic measures how far the sample result is from what \(H_0\) predicts, in standard errors. For a mean with known \(\sigma\), \(z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}\). A large absolute value means the data are far from the null's prediction.
z = \frac{\bar{x} - \mu_0}{\sigma / \sqrt{n}}The p-value
The p-value is the probability, assuming \(H_0\) is true, of getting a result at least as extreme as the one observed. A small p-value means the data would be surprising if \(H_0\) were true, which is evidence against \(H_0\). It is not the probability that \(H_0\) is true.
Significance level and decisions
Choose a significance level \(\alpha\), often 0.05, before seeing the data. If the p-value is at most \(\alpha\), reject \(H_0\): the result is statistically significant. Otherwise, fail to reject \(H_0\). Failing to reject is not proof that \(H_0\) is true, just as 'not guilty' is not proof of innocence.
Worked example
A cereal company claims boxes contain 500 g on average. A sample of 36 boxes has \(\bar{x} = 496\) g, with known \(\sigma = 12\) g. Test \(H_0: \mu = 500\) against \(H_a: \mu < 500\). \(z = \frac{496 - 500}{12/6} = -2\), so the p-value is \(P(Z < -2) \approx 0.023\). At \(\alpha = 0.05\), reject \(H_0\): there is evidence the boxes are underfilled.
Type I and Type II errors
A Type I error rejects a true \(H_0\), a false alarm; its probability is \(\alpha\). A Type II error fails to reject a false \(H_0\), a missed detection; its probability is \(\beta\). Lowering \(\alpha\) reduces false alarms but increases misses, so the choice depends on which error is more costly.
Power
Power is the probability of correctly rejecting a false \(H_0\): \(1 - \beta\). Power increases with larger samples, larger true effects, less variability, and a larger \(\alpha\). Planning a study's sample size for adequate power, often 80%, avoids wasting effort on studies that cannot detect real effects.
Statistical versus practical significance
With huge samples, tiny and unimportant differences become statistically significant. A diet that lowers weight by 0.1 kg might give \(p < 0.001\) in a study of 100,000 people but matter to no one. Always report the size of the effect, ideally with a confidence interval, not just the p-value.
Key takeaways
State \(H_0\) and \(H_a\) about parameters. Compute a test statistic and p-value assuming \(H_0\). Reject \(H_0\) when \(p \le \alpha\); never 'accept' \(H_0\). Balance Type I and Type II errors, plan for power, and judge practical importance alongside significance.
Flashcard
Null hypothesis
Check your understanding
A test gives a p-value of 0.003. At alpha = 0.05, the correct conclusion is: