10 Common Statistical Mistakes (And How to Avoid Them)

Review the reasoning as carefully as the arithmetic

Many statistical errors produce a plausible number. The problem is the question it answers, the assumptions it needs, or the conclusion attached to it. Use these checks when reviewing your own analysis.

1. Reporting a relative change without the baseline

A rate falling from 2% to 1% is a 50% relative reduction and a 1-percentage-point absolute decrease. Report the baseline and both quantities when they serve the interpretation. The denominator is part of the claim.

2. Treating more rows as more independent units

Ten measurements from each of ten people do not automatically provide 100 independent people. Shared individuals, classrooms, shifts, or time periods can create dependence. Identify the unit and use a design-aware method.

3. Reversing a conditional probability

P(positive test | condition) is not P(condition | positive test). The latter also depends on the base rate and false-positive probability. Work through the Bayes lesson or everyday examples before interpreting a fictional screening model.

4. Giving a p-value the wrong meaning

A p-value is conditional on a null model and analysis assumptions. It is not the probability the null is true, the probability chance caused the result, or a measure of importance.

The ASA p-value statement cautions against reducing interpretation to whether a threshold was crossed.

5. Calling nonsignificance “no effect”

A wide interval spanning meaningful benefits and harms leaves the question unresolved. It does not establish equality. Practical equivalence is a separate question requiring predefined acceptable bounds and a suitable analysis.

6. Confusing confidence coverage with individual spread

A 95% interval for a mean does not contain 95% of observations. It reflects a method’s repeated-sampling coverage for the mean under its assumptions. A prediction interval concerns a new observation and is wider under the same normal mean model.

Read confidence intervals for worked interpretations.

7. Letting a normality test choose the whole analysis

Small samples can hide departures; large samples can detect tiny ones. “n ≥ 30” is not a universal guarantee. Study design, the estimand, plots, and influence matter. Rank methods can test a different question from mean methods.

8. Searching until a result is significant

Multiple outcomes, repeated checking, and data-dependent exclusions change the analysis’s error behavior. Twenty independent tests at α = 0.05 have probability about 64.2% of at least one rejection when all nulls are true.

Predefine primary analyses, use suitable multiplicity or sequential methods, and label exploratory findings. Keep a record of what you tried.

9. Deleting an outlier because it changes the result

An unusual point can be an error, a valid rare event, or evidence of a different mechanism. Investigate and document the reason for any exclusion. A preferred p-value is not a reason to remove an observation.

10. Calling association causation

A before-and-after change may reflect an intervention, background trends, practice, or regression to the mean. A regression slope may reflect confounding. A significant test does not distinguish these explanations by itself.

Use causal language only when the design and assumptions support it. Otherwise describe the observed association or estimated change accurately.

A short review before sharing

Write one sentence each for the question, unit of analysis, effect estimate, uncertainty, method, and main limitation. If you cannot do that clearly, revisit the analysis before adding a significance label.

The test-selection guide, hypothesis-testing lesson, and reporting guide connect these checks to usable examples.