Regression Diagnostics

Diagnostics ask whether the fitted model serves its purpose

A fitted line and a high R² do not establish a useful model. Diagnostics compare the observed data with the model’s structure and assumptions. Begin by specifying the goal: description, coefficient inference, prediction or a causal question. Different failures have different consequences.

Residuals are observed minus fitted outcomes, eᵢ = yᵢ − ŷᵢ. They help reveal patterns left unexplained, but they are not the unobserved model errors and are constrained by the fitting process.

Worked example: a curve hidden by one line

Take hypothetical values x = 0, 1, 2, 3, 4 and y = 4, 1, 0, 1, 4. The least-squares straight line has slope 0 and intercept 2. Its residuals are 2, −1, −2, −1, 2.

The U-shaped residual pattern suggests a missing nonlinear relationship. A small correlation here would not mean no relationship. Adding a justified quadratic term can represent this particular curve, but selecting a flexible model to fit five points is not a demonstration of reliable out-of-sample prediction.

Read plots as evidence, not mechanical certificates

Check	Pattern to investigate	Possible next step
Residuals vs fitted values or predictors	Curvature	Reconsider functional form or omitted structure
Residual spread	Funnel or changing scatter	Model variance or consider suitable robust inference
Residuals vs time/group	Runs, cycles, group patterns	Investigate dependence and collection design
Q–Q plot	Systematic tail departures	Assess error model and inference sensitivity
Leverage and influence	Extreme predictors or large fit changes	Check records and sensitivity; explain exclusions

A normal Q–Q plot assesses a distributional approximation, not independence. A patternless plot cannot verify random sampling. The role of normal errors depends on the inference and prediction procedure; it is not a prerequisite for calculating least-squares coefficients.

Separate outliers, leverage and influence

A residual outlier has an unusual outcome relative to its prediction. A high-leverage observation has an unusual predictor configuration. An influential observation materially changes fitted results when perturbed or omitted.

Cook’s distance combines residual and leverage information to assess influence. A high value is not proof of a data error or merely a high-leverage flag. Inspect the original record and compare reasonable analyses with and without it. Do not remove valid observations solely to obtain significance.

With multiple predictors, leverage concerns their joint configuration; a point can be unusual in combination even if each individual predictor is within a familiar range.

Check predictions outside the fitting data

Training residuals can look reassuring after overfitting. Evaluate on held-out data or with an appropriate validation design. For repeated users, split by user when predicting new users; for future forecasts, preserve time order. Avoid fitting preprocessing on held-out outcomes.

Report prediction errors in meaningful units and compare a simple baseline. Confidence intervals for mean predictions differ from prediction intervals for new individual observations; the latter include individual variation and are generally wider under the same model.

Diagnostics cannot identify a causal effect by themselves. Model adjustments, clean residuals and strong prediction do not replace a defensible study design.

Practice

An observation has unusual predictor values but sits close to the fitted line. Must it be a residual outlier?

Show the explanation

No. It may have high leverage and a small residual. Investigate influence separately: unusual predictors can affect the fit even when the residual is small. Check validity and sensitivity rather than automatically deleting it.

Continue

Review linear regression and multiple regression. NIST explains regression model validation and graphical checks.