Connecting estimation and decisions
Confidence intervals and tests are two views of the same evidence. Used together, they support better decisions.
Intervals contain plausible values
Every value inside a 95% interval is a plausible parameter value, in the sense that a two-sided test at \(\alpha = 0.05\) would not reject it. Values outside are implausible. So an interval answers many tests at once, and also shows how big the effect could be.
Precision and decisions
Suppose a new medicine's interval for reducing blood pressure is 1 to 15 mmHg. The effect is significant, but it could be tiny or substantial. A larger study might narrow it to 7 to 10, enough to make a confident medical decision. Precision, not just significance, guides action.
One-sided intervals
Sometimes only one bound matters: a regulator may need to be confident that a contaminant's mean level is below a limit. A one-sided upper confidence bound answers that directly and pairs with a one-sided test.
Planning studies
Before collecting data, decide the margin of error or power you need and compute the required sample size. Under-powered studies waste resources and add to the replication crisis; over-sized studies may be unethical in medicine if they expose more people than necessary to an inferior treatment.
Worked example: an A/B test
An online store tests two checkout designs. Design B's purchase rate is 2.1 percentage points higher, with a 95% interval of 0.4 to 3.8 points. Since the interval excludes 0, B is very likely better, and the lower bound tells the business the smallest plausible gain to plan around.
Communicating uncertainty
Present intervals with plain language: 'Our best estimate is 27%, and the true value is likely between 24% and 30%.' Visuals like error bars help, but always state what the bars represent, since some show standard errors and others confidence intervals.
Uncertainty beyond sampling
Intervals capture sampling error, not measurement error, model mistakes, or biased data. Good practice combines statistical uncertainty with judgement about these other sources, and reports limitations honestly.
Common mistakes
Treating overlapping error bars as proof of no difference. Ignoring the width of an interval when making decisions. Forgetting to plan sample size. Presenting estimates without any uncertainty.
Key takeaways
Intervals list plausible values and show effect size and precision. They agree with two-sided tests. Plan sample sizes for adequate precision or power. Communicate uncertainty clearly and remember sources of error that intervals do not capture.
Flashcard
Values inside a confidence interval
Check your understanding
A 95% CI for a difference in success rates is (0.4, 3.8) percentage points. What can we conclude?