Residuals and model fit
A regression line is only useful if it fits. Residuals and r-squared tell you how well, and residual plots reveal when a straight line is the wrong model.
Residuals
A residual is observed minus predicted: \(e = y - \hat{y}\). Positive residuals mean the point is above the line; negative, below. The least-squares residuals always sum to zero. Residuals are the 'leftover' variation the line does not explain.
e = y - \hat{y}Residual plots
Plot residuals against \(x\) (or against \(\hat{y}\)). A good linear model leaves a random scatter around zero with no pattern. A curved pattern means the relationship is not linear; a fan shape (spread growing with \(x\)) means the variability is not constant. Residual plots often reveal problems the original scatterplot hides.
The coefficient of determination
\(r^2\) is the fraction of the variation in \(y\) explained by the linear relationship with \(x\). If \(r = 0.8\), \(r^2 = 0.64\): 64% of the variation in prices is explained by size, and 36% is due to other factors. \(r^2\) is between 0 and 1 and is often reported as a percentage.
r^2 = 1 - \frac{\sum e^2}{\sum (y - \bar{y})^2}The standard deviation of residuals
The residual standard deviation \(s_e\) gives the typical size of prediction errors, in the units of \(y\). If \(s_e = 36\) thousand dollars, a typical prediction misses by about \$36,000. Report \(s_e\) alongside \(r^2\) for a complete picture of fit.
Outliers and influential points
An outlier in regression has a large residual. An influential point changes the slope substantially when removed, often because it has an extreme \(x\) value (high leverage). Fit the model with and without such points to see how much they matter, and investigate why they are unusual.
Transformations
If the residual plot is curved, transform a variable. Exponential growth becomes linear after taking \(\ln y\); power relationships become linear after taking logs of both variables. Then fit a straight line to the transformed data and transform predictions back.
Worked example
A line predicting fuel efficiency from car weight has \(r^2 = 0.75\) and \(s_e = 2.1\) mpg, but the residual plot shows a curve: residuals are positive at both ends and negative in the middle. The linear model systematically misses; a log transformation of weight or a curved model would fit better, despite the decent \(r^2\).
Common mistakes
Judging fit by \(r^2\) alone without a residual plot. Forgetting that a high \(r^2\) does not mean causation. Deleting outliers without justification. Comparing \(r^2\) across models with different response variables.
Key takeaways
Residuals are observed minus predicted and should look random in a residual plot. \(r^2\) is the fraction of variation explained; \(s_e\) is the typical prediction error. Check for outliers and influential points, and transform variables when the pattern is curved.
Flashcard
y - \hat{y}
Check your understanding
If r = 0.9 between study time and score, what percent of variation in scores is explained by study time?