The least-squares line
When two quantitative variables have a linear relationship, a regression line summarises it and lets us predict one variable from the other.
The regression line
A regression line \(\hat{y} = b_0 + b_1x\) predicts the response \(y\) from the explanatory variable \(x\). The slope \(b_1\) is the predicted change in \(y\) for a one-unit increase in \(x\); the intercept \(b_0\) is the predicted \(y\) when \(x = 0\), which may or may not be meaningful.
\hat{y} = b_0 + b_1 xLeast squares
The least-squares line minimises the sum of squared vertical distances between the points and the line. Its slope is \(b_1 = r\frac{s_y}{s_x}\), and it always passes through \((\bar{x}, \bar{y})\), so \(b_0 = \bar{y} - b_1\bar{x}\). The connection to correlation shows that a stronger correlation gives a steeper standardised slope.
b_1 = r\,\frac{s_y}{s_x}Worked example
For a set of houses, size \(x\) (hundreds of square feet) has \(\bar{x} = 20\), \(s_x = 5\); price \(y\) (thousands of dollars) has \(\bar{y} = 300\), \(s_y = 60\); and \(r = 0.8\). Then \(b_1 = 0.8 \cdot \frac{60}{5} = 9.6\) and \(b_0 = 300 - 9.6(20) = 108\). The line is \(\hat{y} = 108 + 9.6x\): each extra 100 square feet adds about \$9,600 to the predicted price.
Interpreting slope and intercept
Interpret the slope with units: 'For each additional hour studied, the predicted exam score increases by 4.2 points.' The intercept here, 108 (thousand dollars) for a house of size 0, is not meaningful, since no house has zero size. Only interpret the intercept when \(x = 0\) is realistic.
Prediction and extrapolation
Use the line to predict \(y\) for \(x\) values within the range of the data. Extrapolation, predicting far outside that range, is risky because the relationship may change. A line fitted to children's heights from ages 2 to 10 would predict absurd heights for 40-year-olds.
Regression toward the mean
Because \(|r| < 1\), predictions are less extreme than the inputs in standardised terms: an individual \(k\) standard deviations above average in \(x\) is predicted to be only \(rk\) standard deviations above average in \(y\). Very tall parents tend to have tall children, but on average less extreme. This is regression toward the mean, and it explains many 'sophomore slumps'.
Regression is not symmetric
The line predicting \(y\) from \(x\) is different from the line predicting \(x\) from \(y\). Always choose the response variable deliberately: the variable you want to predict goes on the \(y\)-axis.
Common mistakes
Interpreting the slope as a causal effect from observational data. Extrapolating far beyond the data. Forgetting units. Using a linear model when the scatterplot is curved.
Key takeaways
The least-squares line minimises squared residuals, has slope \(r s_y/s_x\), and passes through the means. Interpret slope with units and the intercept only when meaningful. Avoid extrapolation, remember regression toward the mean, and choose the response variable carefully.
Flashcard
Regression line
Check your understanding
For the line y-hat = 12 + 3.5x, what does 3.5 mean?