Multiple Regression

Model a mean conditional on several predictors

Multiple linear regression represents the conditional mean of a numerical outcome using several predictors:

$E[Y\mid X]=\beta_0+\beta_1X_1+\cdots+\beta_kX_k.$

A fitted coefficient describes the model change in mean outcome for a one-unit predictor change holding the other included predictors fixed. This conditional comparison may differ from a simple two-variable association. It is not automatically causal.

Worked example: read an adjusted model

Suppose an illustrative fitted model predicts delivery duration in hours:

$\hat y=2+0.1(\text{distance in km})+1.5(\text{express indicator}).$

The express indicator is 1 for express, 0 otherwise. At distance 20 km, the standard prediction is 4 hours, and the express prediction is 5.5 hours. The coefficient 1.5 compares express with standard at the same model distance. It does not show that selecting express adds 1.5 hours causally: urgent or difficult deliveries may preferentially use express.

The intercept predicts 2 hours at distance 0 for standard shipping. If zero distance is outside the observed data, that intercept may have little practical meaning. Predictions should stay within a defensible range and population.

Coding and interactions change interpretation

A categorical predictor with three levels commonly uses two indicators relative to a stated reference level. Treating codes 1, 2, 3 as a single numerical predictor imposes equal spacing and a linear trend that may not be justified.

If the model adds 0.05 × distance × express, then the express difference is 1.5 + 0.05 × distance. At 20 km it is 2.5 hours, not the coefficient 1.5 alone. The distance slope is 0.1 for standard and 0.15 for express. Interpret an interaction using the full expression and useful predictor values.

Transformations also change units. In a log-outcome model, exponentiating a fitted log mean is not generally the same as estimating the arithmetic mean on the original scale. State the prediction target and any correction used.

Collinearity concerns information, not automatic deletion

Highly related predictors can make individual coefficients unstable and uncertain. They may still support useful joint prediction. Check redundancy, uncertainty and the study question before removing a variable. A variance inflation factor is a diagnostic, not a universal pass/fail threshold.

Adding predictors tends to increase ordinary training R², even when they add little useful signal. Adjusted R² applies a complexity penalty but does not replace out-of-sample evaluation. Use validation appropriate to the intended use, preserving user groups or time order where needed.

Check the design and the residual model

Inspect residual plots for systematic curvature, changing variance and influential observations. Independence is a property to investigate through collection design and temporal or group structure; a patternless residual plot does not prove it.

Classical standard errors and small-sample tests need additional error assumptions. Robust covariance estimates address some variance misspecification but do not repair omitted-variable bias, dependence or an incorrect causal claim.

Practice

In a model containing x, group and x × group, can the group coefficient alone describe the group difference for every x?

Show the explanation

No. It describes the difference at x = 0 (or at the chosen centered zero). The interaction makes the group difference vary with x. Report differences at meaningful values and acknowledge uncertainty.

Continue

Check regression diagnostics, then study logistic regression for binary outcomes. NIST covers model fitting and model validation.