Simple Linear Regression

Model a mean response as a function of a predictor

Simple linear regression relates a numerical response Y to one predictor X. The model is:

$Y_i=\beta_0+\beta_1x_i+\varepsilon_i$

The fitted line, ŷ = b₀ + b₁x, estimates the mean response at a predictor value. An individual outcome also has error around that line.

The slope has units of Y per unit of X. It describes an association under the model; it is not automatically the causal effect of changing X.

Fit a line from raw data

Use this small hypothetical dataset:

Study hours x	Exam score y
2	65
4	75
6	80
8	88
10	92

The means are x̄ = 6 and ȳ = 80. Define Sxx = Σ(x − x̄)² and Sxy = Σ(x − x̄)(y − ȳ).

x − x̄	y − ȳ	Product	Squared x deviation
−4	−15	60	16
−2	−5	10	4
0	0	0	0
2	8	16	4
4	12	48	16
Total		134	40

$b_1=\frac{S_{xy}}{S_{xx}}=3.35,\qquad b_0=\bar y-b_1\bar x=59.9$

The fitted line is predicted score = 59.9 + 3.35 × hours. Least squares chooses these coefficients to minimize the sum of squared residuals.

Download the example pairs (CSV)

Interpret without overclaiming

Within the observed 2–10 hour range, one additional study hour is associated with a 3.35-point higher fitted mean score. This does not prove that making someone study an extra hour raises their score by that amount. Prior knowledge, motivation, and other factors may influence both variables.

The intercept 59.9 is the fitted response at zero hours. Since zero is outside the observed range, a literal prediction there is extrapolation. The intercept can be mathematically necessary without a useful interpretation.

At seven hours, the fitted mean is 59.9 + 3.35 × 7 = 83.35 points. This is interpolation. It is still uncertain and should not be reported as a guaranteed individual score.

Residuals show what the line misses

A residual is observed minus fitted: eᵢ = yᵢ − ŷᵢ.

Hours	Observed score	Fitted score	Residual
2	65	66.60	−1.60
4	75	73.30	1.70
6	80	80.00	0
8	88	86.70	1.30
10	92	93.40	−1.40

SSE, the residual sum of squares, is 9.1. With an intercept, least-squares residuals sum to zero up to numerical precision. That algebraic property does not establish normality or independence.

The residual standard error is:

$s_e=\sqrt{\frac{SSE}{n-2}}=\sqrt{\frac{9.1}{3}}=1.7416\text{ points}$

It estimates the model’s error SD under its assumptions. It is not the mean absolute residual or the standard error of the slope.

R² describes the fit to these data

The total corrected sum of squares is SST = Σ(y − ȳ)² = 458. Thus:

$R^2=1-\frac{9.1}{458}=0.9801$

About 98.01% of the response’s sample variation around its mean is accounted for by this fitted line. For simple least-squares regression with an intercept, R² equals Pearson r²; here r ≈ 0.9900.

A high R² does not guarantee unbiased predictions on new data, a correct model, or a causal relationship. These five points are deliberately simple teaching data. Real model assessment needs more evidence and, for predictive work, validation outside the fitting data.

Inference requires more than fitting a line

For classical small-sample slope tests and intervals, assume a correct linear conditional mean, independent errors, constant error variance, and normal errors. X itself does not need to be normally distributed.

The estimated slope SE is:

$SE(b_1)=\frac{s_e}{\sqrt{S_{xx}}}=0.2754$

For H₀: β₁ = 0, t = 3.35/0.2754 ≈ 12.1651, df = 3, two-sided p ≈ 0.0012. A 95% interval is 3.35 ± 3.1824 × 0.2754, approximately [2.4736, 4.2264] points per hour.

This interval reflects uncertainty under the model; it does not remove confounding. Inspect residuals versus fitted values for curves and changing spread, examine influential observations, and use study design to assess independence.

Mean-response intervals and prediction intervals

A confidence interval for the mean response at x₀ quantifies uncertainty about E(Y | X = x₀). A prediction interval for a new individual adds the individual’s error variability:

$SE_{mean}=s_e\sqrt{\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}}$

$SE_{prediction}=s_e\sqrt{1+\frac1n+\frac{(x_0-\bar x)^2}{S_{xx}}}$

The prediction interval is wider under the same model and confidence level. Both widen as x₀ moves away from x̄; neither formula makes far-out extrapolation trustworthy.

Practice

Given r = 0.8, sx = 4, sy = 10, x̄ = 20, and ȳ = 50, find the fitted line. What does its slope mean?

Show the solution

b₁ = r(sy/sx) = 0.8 × 10/4 = 2. b₀ = 50 − 2 × 20 = 10. The line is ŷ = 10 + 2x. One unit higher X is associated with a two-unit higher fitted mean Y. The slope’s units and causal interpretation depend on the variables and design.

References and next steps

NIST’s linear least-squares reference describes the model and fitting method. Continue to regression diagnostics and multiple regression. Use the correlation calculator to check r for the example; it does not fit a regression line or calculate prediction intervals.