Inference for regression and multiple regression
Sample regression lines vary from sample to sample. Inference tells us whether a relationship is real, and multiple regression handles several predictors at once.
The population regression model
We imagine a true line \(\mu_y = \beta_0 + \beta_1x\) for the population, with individual responses scattering around it. The sample slope \(b_1\) estimates \(\beta_1\). Conditions: a linear pattern, independent observations, roughly normal residuals, and equal spread of residuals across \(x\) (remember LINE).
Testing the slope
To test whether there is any linear relationship, test \(H_0: \beta_1 = 0\) with \(t = \frac{b_1}{SE_{b_1}}\) and \(n - 2\) degrees of freedom. A small p-value means the slope is significantly different from zero. Software reports \(SE_{b_1}\), \(t\), and the p-value in every regression table.
t = \frac{b_1}{SE_{b_1}},\ df = n - 2A confidence interval for the slope
\(b_1 \pm t^* SE_{b_1}\) gives a range of plausible slopes. If a study finds the slope of exam score on study hours is 4.2 with a 95% interval of 2.5 to 5.9 points per hour, we are fairly confident each additional hour is associated with a gain somewhere in that range.
b_1 \pm t^* SE_{b_1}Reading regression output
A typical table lists each coefficient with its estimate, standard error, t statistic, and p-value, plus \(r^2\) and the residual standard error. Learning to read this table lets you understand research papers, business reports, and machine learning summaries.
Multiple regression
Real outcomes depend on several variables. Multiple regression fits \(\hat{y} = b_0 + b_1x_1 + b_2x_2 + \cdots\). Each coefficient is the predicted change in \(y\) for a one-unit change in that variable, holding the others constant. This lets researchers adjust for confounders, such as estimating the effect of education on income while holding experience fixed.
\hat{y} = b_0 + b_1x_1 + b_2x_2 + \cdotsCategorical predictors
Categories enter a regression through indicator (dummy) variables equal to 1 for one group and 0 otherwise. The coefficient then measures the average difference between that group and a baseline group, adjusting for the other predictors.
Multicollinearity and overfitting
When predictors are highly correlated with each other, their individual coefficients become unstable, a problem called multicollinearity. Adding too many predictors fits the sample's noise, called overfitting, so the model predicts new data poorly. Adjusted \(r^2\) and testing on new data help guard against this.
Common mistakes
Interpreting a multiple regression coefficient without 'holding others constant'. Claiming causation from observational regression. Ignoring the LINE conditions. Adding predictors just to raise \(r^2\).
Key takeaways
Test \(H_0: \beta_1 = 0\) with \(t = b_1/SE_{b_1}\) on \(n - 2\) degrees of freedom, and give a confidence interval for the slope. Check LINE conditions. Multiple regression estimates each predictor's association holding others constant. Watch for multicollinearity and overfitting.
Flashcard
Population slope
Check your understanding
A regression of income on education and experience gives an education coefficient of 3.2 (thousand dollars per year of education). What does it mean?