Residuals are the differences between observed values and values predicted by the model:
\[e_i = y_i - \hat{y}_i\]
They represent the part of the data not explained by the model—the “leftover” variation. Analyzing residuals helps us check whether the assumptions of our model are met and identify potential problems.
A regression model might fit the data well according to R-squared while still being inappropriate. The model might capture the wrong pattern, miss non-linear relationships, or be unduly influenced by outliers. Residual analysis reveals these problems.
Remember Anscombe’s Quartet—four datasets with identical regression lines but very different patterns. Looking only at the regression output would miss these differences entirely.
If the relationship is truly linear, residuals should show no systematic pattern when plotted against fitted values or the predictor variable. A curved pattern suggests non-linearity.
# Create data with non-linear relationship
set.seed(42)
x <- seq(0, 10, length.out = 100)
y <- 2 + 0.5 * x + 0.1 * x^2 + rnorm(100, sd = 1)
model <- lm(y ~ x)
par(mfrow = c(1, 2))
plot(x, y, main = "Data with Quadratic Pattern")
abline(model, col = "red")
plot(fitted(model), residuals(model),
main = "Residuals vs Fitted",
xlab = "Fitted values", ylab = "Residuals")
abline(h = 0, col = "red", lty = 2)
The curved pattern in the residual plot reveals that a linear model is inadequate.
Residuals should be approximately normally distributed. Check with a histogram or Q-Q plot:
# Good model for comparison
x2 <- rnorm(100)
y2 <- 2 + 3 * x2 + rnorm(100)
good_model <- lm(y2 ~ x2)
par(mfrow = c(1, 2))
hist(residuals(good_model), breaks = 20, main = "Histogram of Residuals",
xlab = "Residuals", col = "lightblue")
qqnorm(residuals(good_model))
qqline(residuals(good_model), col = "red")
Points on the Q-Q plot should fall approximately along the diagonal line. Systematic departures indicate non-normality.
Residuals should have constant variance across the range of fitted values. A fan or cone shape indicates heteroscedasticity (unequal variance).
Residuals should be independent of each other. This is hard to check visually but is violated when observations are related (e.g., repeated measurements on the same subjects, or time series data).
R provides built-in diagnostic plots for linear models:
# Standard diagnostic plots
par(mfrow = c(2, 2))
plot(good_model)
These four plots show: 1. Residuals vs Fitted: Check for linearity and homoscedasticity 2. Q-Q Plot: Check for normality 3. Scale-Location: Check for homoscedasticity 4. Residuals vs Leverage: Identify influential points
Not all observations affect the regression equally. Leverage measures how unusual an observation’s X value is—points with extreme X values have more potential to influence the fitted line.
Cook’s Distance measures how much the regression would change if an observation were removed. High Cook’s D values indicate influential points that merit closer examination.
# Check for influential points
influence.measures(good_model)$is.inf[1:5,] # First 5 observations dfb.1_ dfb.x2 dffit cov.r cook.d hat
1 FALSE FALSE FALSE FALSE FALSE FALSE
2 FALSE FALSE FALSE FALSE FALSE FALSE
3 FALSE FALSE FALSE FALSE FALSE FALSE
4 FALSE FALSE TRUE FALSE FALSE FALSE
5 FALSE FALSE FALSE FALSE FALSE FALSEWhen assumptions are violated, several approaches may help:
Transform the data: Log, square root, or other transformations can stabilize variance and improve linearity.
Use robust regression: Methods like rlm() from the MASS package down-weight influential observations.
Try a different model: Non-linear regression, generalized linear models, or generalized additive models may be more appropriate.
Remove outliers: Only if you have substantive reasons—never simply to improve fit.
A systematic approach to residual analysis:
Residual analysis is not optional—it is an essential part of any regression analysis. Models that look good on paper may tell misleading stories if their assumptions are violated.