How to Choose the Right Statistical Test

Write the question in the parameter’s language

“Which test should I use?” is easier to answer after “What do I want to estimate or compare?” A mean wait, the probability of a signup, a median change, and a relationship between categories are different targets.

Start by writing the population, outcome, independent unit, and desired comparison. Then select a method that addresses that comparison under a defensible model. The interactive test guide gives common starting points; this article explains the decisions behind them.

1. Identify the outcome

• Numerical measurements can support mean or distributional comparisons.
• Binary outcomes produce success proportions.
• Multiple categories produce counts in category or contingency tables.
• Ordered categories may support rank or ordinal-model questions.

An identifier stored as a number is still a label. A five-point rating also does not automatically have equal distances between categories.

2. Identify what is independent

Different people assigned to two conditions are usually an independent-group design. The same people measured twice are paired. People within the same class, household, or clinic may share conditions; measurements over time can be correlated.

The number of rows is not necessarily the number of independent units. A thousand sessions from a hundred users are not automatically a thousand independent people.

3. Match the method to the target

Target and design	Common starting method	Check
One mean versus a target, unknown σ	One-sample t	Independence, skew, extreme influence
Two independent means	Welch t	Independent groups and suitable mean inference
Mean within-pair change	Paired t	Independent pairs and distribution of differences
Several independent means	ANOVA or Welch ANOVA	Model, variance assumptions, follow-up comparisons
Fixed category probabilities	Chi-square goodness of fit	Specified probabilities and expected counts
Association of two categories	Chi-square independence or an exact method	Independent units and table sparsity
Linear numerical association	Pearson correlation	Scatter plot, influential points, inferential model
Mean response versus predictors	Regression	Functional form, dependence, variance, influence

A one-sample proportion test uses a binomial model or suitable approximation, not a t-test of arbitrary category labels. A paired binary outcome needs a paired method, not an independent 2×2 analysis.

4. Treat assumptions as model questions

A normality test is not a universal switch between parametric and non-parametric procedures. Large samples can flag tiny departures; small samples may fail to reveal important ones. Plot the relevant variable or differences, inspect extremes, and evaluate independence from the design.

There is no general rule “n ≥ 30 means use z.” Mean inference with σ estimated from the sample usually uses t. Rank tests can be useful, but may address distributional or rank questions rather than the mean or median you intended. Read non-parametric tests before substituting one mechanically.

5. Plan the analysis before results steer it

Specify the primary outcome, groups, exclusions, test direction, significance threshold, and stopping rule. Multiple comparisons and repeated checking require suitable procedures. Report exploratory discoveries as exploratory.

Choose sample size based on meaningful precision or a specified effect and power target, not on a plan to keep collecting until p < 0.05.

A worked selection example

A service wants to compare order wait under two workflows. Each order has a numerical wait, but workflows are assigned by shift. Orders within a shift share demand, staffing, and conditions.

An ordinary order-level Welch test may ignore this clustering. The design may require shift-level comparison or a model that accounts for shift structure. A calculator cannot decide that from two pasted columns.

If the workflows were independently assigned to suitable units and the outcome model was appropriate, an independent mean comparison could be a reasonable starting point. Design determines that suitability before arithmetic begins.

Report the answer, not only the procedure

A useful result includes the estimated effect in original units, uncertainty, sample sizes and units, method, assumptions, and limitations. Statistical significance does not establish causality or practical value. A nonsignificant result does not establish equivalence.

The ASA p-value statement emphasizes interpretation beyond a threshold. Read the confidence interval lesson, t-test lesson, or chi-square lesson for worked calculations.