Two-Sample Tests

Two groups are not always two independent samples

Choose a method from the outcome and the relationship between observations:

Outcome	Independent groups	Paired observations
Numerical quantity, mean comparison	Welch two-sample t-test	Paired t-test on differences
Binary outcome	Suitable two-proportion method or 2×2 table analysis	Paired binary method such as McNemar’s
Ordinal/rank-based question	Mann–Whitney with appropriate interpretation	Signed-rank or sign test, depending on assumptions

Pairing is determined by identity or matching, not by two columns having equal lengths. Clustered samples and repeated measurements can require models beyond these simple choices.

Independent mean differences

Welch’s statistic for H₀: μ₁ − μ₂ = 0 is:

$t=\frac{\bar x_1-\bar x_2}{\sqrt{s_1^2/n_1+s_2^2/n_2}}$

Use its Welch–Satterthwaite degrees of freedom, not n₁ + n₂ − 2 unless using a justified pooled equal-variance model. Keep group order consistent: reversing groups changes the difference and t signs, not a two-sided p-value.

The t-test lesson gives the full formula, a worked independent-group example, and a paired example that can be entered into the calculator. This lesson extends the comparison to proportions.

Paired mean differences

For each matched unit, calculate dᵢ = first measurement − second measurement. Then use a one-sample t-test of those differences:

$t=\frac{\bar d-\delta_0}{s_d/\sqrt n},\qquad df=n-1$

Here n counts complete independent pairs. The relevant small-sample normality condition concerns the differences, not the two marginal distributions separately.

A before-and-after change can reflect the intervention, a background trend, practice, or regression to the mean. A paired p-value alone cannot distinguish those mechanisms. Include an appropriate control or randomized design when the question is causal.

Compare two independent proportions

Suppose option A converts 30 of 100 independent users and option B converts 45 of 100. Treat one user as one unit, with assignment independent across units.

Observed proportions are p̂_A = 0.30 and p̂_B = 0.45. The difference A minus B is −0.15, or −15 percentage points. B’s relative increase over A is 0.15/0.30 = 50%; that is a different quantity.

For H₀: p_A = p_B, pool the successes to estimate their common null probability:

$\hat p_{pool}=\frac{x_A+x_B}{n_A+n_B}=\frac{75}{200}=0.375$

$SE_0=\sqrt{\hat p_{pool}(1-\hat p_{pool})\left(\frac1{n_A}+\frac1{n_B}\right)}=0.06847$

Thus z = −0.15/0.06847 ≈ −2.1909 and the uncorrected two-sided normal p-value is 0.0285. Null expected successes and failures in both groups are large enough for a common introductory normal-approximation check.

This uncorrected two-proportion test corresponds to the uncorrected Pearson 2×2 chi-square test, with χ² = z². Continuity-corrected or exact methods can produce different results.

The interval uses a different standard error

A large-sample Wald interval for the proportion difference uses the unpooled estimated SE:

$SE_{CI}=\sqrt{\frac{\hat p_A(1-\hat p_A)}{n_A}+\frac{\hat p_B(1-\hat p_B)}{n_B}}$

Here SE_CI ≈ 0.06764, so the approximate 95% interval is −0.15 ± 1.96 × 0.06764 = [−0.2826, −0.0174]. Other interval methods can improve coverage, especially near boundaries or with small samples. Do not assume the pooled test SE and interval SE are interchangeable.

This supports a difference under the specified design and approximation. Acting on it also requires consideration of the effect size, costs, representativeness, and whether the analysis was preplanned.

Common failures in comparisons

• Repeated checking: fixed-sample p-values do not retain their intended interpretation when you keep checking and stop at significance without a suitable sequential design.
• Wrong unit: sessions, clicks, or purchases can repeat within a user. Independence may need a user-level analysis or a clustered method.
• Selective outcomes: trying many metrics and reporting only the smallest p-value hides multiplicity.
• Overlapping intervals: overlap of separate group intervals is not a test of their difference.
• Mismatched pairing: losing row identity changes a paired analysis completely.

Practice

A study measures the same 40 people before and after a change. Should the paired mean test use df = 79 or df = 39?

Show the explanation

Use 39, based on 40 differences. Eighty measurements do not create 80 independent people. Check the distribution of the differences and the independence of the pairs.

Continue learning

Use the t-test calculator for raw-data mean comparisons. Read chi-square tests for independent categorical counts and sample size determination for planning the comparison before collecting data.