T-Tests: Comparing Means

Choose the design before choosing the formula

T-tests concern population means when variability is estimated from the data. They share the idea “estimated difference divided by its standard error,” but the standard error and degrees of freedom depend on the design.

Design	Question	Unit of analysis
One sample	Does one population mean differ from a specified value?	Independent observation
Two independent samples	Do two population means differ?	Independent observation within a group
Paired	Does a population mean within-pair difference differ from zero?	Independent pair

Two lists of the same length do not automatically form paired data. Pairing comes from the study: the same person measured twice or a defensible matching scheme.

One-sample t-test from raw data

Five recorded values are 12, 14, 16, 18, 20. Test H₀: μ = 15 against H₁: μ ≠ 15.

The sample mean is 16. Squared deviations from 16 sum to 40, so sample variance is 40/4 = 10 and s = 3.1623.

$SE=\frac{s}{\sqrt5}=1.4142,\qquad t=\frac{16-15}{1.4142}=0.7071$

With df = 4, the two-sided p-value is 0.5185. At α = 0.05, fail to reject H₀. The 95% interval for the mean is 16 ± 2.7764 × 1.4142, or about [12.0735, 19.9265].

The interval is wide because the sample is small. “No evidence of a difference at this threshold” does not demonstrate that μ = 15.

Reproduce it in the t-test calculator: one-sample mode, values 12,14,16,18,20, hypothesized mean 15. The site calculates two-sided tests.

Two independent groups: Welch’s test

Suppose group A has n₁ = 30, mean 78, SD 10; group B has n₂ = 35, mean 82, SD 12. Assume independent observations within and between groups and a suitable distributional model.

Welch’s test does not assume equal population variances:

$SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}},\qquad t=\frac{\bar{x}_1-\bar{x}_2}{SE}$

$\nu=\frac{(s_1^2/n_1+s_2^2/n_2)^2}{(s_1^2/n_1)^2/(n_1-1)+(s_2^2/n_2)^2/(n_2-1)}$

Here SE ≈ 2.7290, t ≈ −1.4657, and ν ≈ 62.96. The two-sided p-value is about 0.1477. The difference A minus B is −4 points, with a 95% interval about [−9.454, 1.454].

Do not round the df to an integer in software unnecessarily. For a printed table with no matching row, a lower df gives a conservative critical value. Software can calculate the fractional-df value directly.

The pooled test uses a common-variance estimate and df = n₁ + n₂ − 2. Its equal-variance assumption needs justification; a nonsignificant variance test is not proof of equality. Welch is a useful default for comparing independent means, not a cure for dependence, severe outliers, or an unsuitable outcome.

NIST’s two-sample t-test reference gives the pooled and Welch formulas and degrees of freedom.

Paired t-test: analyze the differences

Suppose five units are measured twice:

Unit	First	Second	First − second
1	12	10	2
2	14	11	3
3	16	15	1
4	18	15	3
5	20	18	2

The differences are 2, 3, 1, 3, 2. Their mean is 2.2, sample SD is 0.8367, and SE is 0.3742.

$t=\frac{\bar d-0}{s_d/\sqrt n}=\frac{2.2}{0.3742}=5.8797$

With df = 4, the two-sided p-value is 0.0042 and the 95% interval for the mean difference is approximately [1.1611, 3.2389].

Enter the first and second columns in paired mode of the calculator. It uses sample 1 minus sample 2. Reversing them changes the signs of t, the mean difference, and the interval, but not a two-sided p-value.

Pair rows by identity before calculating. Sorting the two columns separately destroys the pairing. Missing one member of a pair must be handled consistently and reported.

A before-and-after difference is evidence of change. Without an adequate comparison design, it does not establish that an intervention caused the change: time trends, practice, and regression to the mean can also contribute.

Assumptions and what to inspect

Independence: justified by sampling or assignment, not by a normality test. For paired tests, pairs should be independent of other pairs.
Relevant distribution: one-sample outcomes, each independent group’s outcomes, or within-pair differences. For exact small-sample t inference, these follow a normal model.
Outliers and skew: plot the relevant values. Larger samples can improve robustness, but “n ≥ 30” is not a guarantee.
Measurement: a numerical outcome for which a mean difference answers the question.

Repeated measures with more than two conditions, clustered observations, and serial measurements generally need methods that model that structure. Do not turn every available row into an independent replicate.

Rank-based alternatives may be useful, but can test different hypotheses. Mann–Whitney is not automatically a test of means, and Wilcoxon signed-rank adds a symmetry assumption for a location interpretation. Choose based on the estimand and design; see non-parametric tests.

Report more than a p-value

Report group sizes, units, mean difference, confidence interval, test version, t, df, and p. Add a practical interpretation and limitations.

For a paired standardized effect, d_z = d̄/s_d. In the five-pair example, d_z ≈ 2.6295. Label this denominator: other paired effect-size definitions exist and are not numerically interchangeable. A large standardized effect in five illustrative pairs does not establish importance or causality.

Cohen’s 0.2/0.5/0.8 conventions are rough benchmarks, not universal rules about practical value. The original units and a meaningful threshold are often more useful.

Practice

A sample has mean 3.2, SD 0.5, n = 16. Test μ = 3.0, two-sided. Use critical t = 2.1314.

Show the calculation and conclusion

SE = 0.5/4 = 0.125; t = 0.2/0.125 = 1.6; df = 15. Since 1.6 < 2.1314, fail to reject at α = 0.05. The 95% interval is 3.2 ± 0.2664, or [2.9336, 3.4664]. It includes 3.0 but does not prove equality.

Next steps

Read the paired t-test walkthrough, use the t-table to connect p-values and critical values, or learn ANOVA for comparing several independent group means.