Effect Size and Statistical Power

Magnitude, uncertainty, and detectability are different

A p-value assesses compatibility with a specified null model. An effect estimate describes the magnitude or pattern observed. Power describes how often a planned test would reject under a specified alternative model.

None of these quantities replaces the others. A tiny difference can be precisely estimated in a large sample; a substantial difference can remain uncertain in a small one. Practical value depends on the setting and original units, not just a significance threshold or standardized label.

Start with the original measurement scale

Suppose two groups have mean scores 75 and 70. The observed difference is 5 points. That is often the most interpretable starting estimate.

Ask what five points means: meaningful improvement, measurement noise, a threshold crossing, or too small to justify a cost? Add an appropriate uncertainty interval and explain the design. An observed association does not establish that group assignment caused it.

For proportions, distinguish an absolute difference in percentage points from a relative change. A rate increasing from 30% to 45% changes by 15 percentage points and by 50% relative to the baseline.

Standardized mean differences need a named denominator

For two independent groups, a common descriptive Cohen’s d divides the mean difference by a pooled sample SD:

$d=\frac{\bar x_1-\bar x_2}{s_p},\qquad s_p^2=\frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}$

If both groups have SD 10 and means 75 and 70, d = 0.5. The standardized difference is half the pooled SD. It does not mean half of participants improved or that the effect is necessarily worthwhile.

For paired data, d_z = d̄/s_d is one definition. Other paired standardized effects use different denominators; label the definition rather than treating all “paired Cohen’s d” values as interchangeable. Small-sample standardized estimates can be biased; Hedges-type corrections are one approach.

The conventional 0.2, 0.5, and 0.8 labels are rough reference points. Their practical meaning is context-dependent. Report the original-unit estimate too.

Other outcomes use other effect measures

Setting	Examples of effect summaries	Interpretation caution
Binary outcome	Risk difference, risk ratio, odds ratio	Odds ratio is not generally a risk ratio
Linear association	Pearson r	Does not capture every nonlinear relationship
One-way ANOVA	η², ω², planned mean contrasts	Model fit does not establish causal explanation
Categorical association	Cramér’s V, category proportions	Practical meaning depends on context and dimensions

For a 2% versus 1% risk, the risk ratio is 2. The odds ratio is (0.02/0.98)/(0.01/0.99) ≈ 2.0204. They are close here because the risks are small; that is not true universally.

Define power for a particular alternative

$\text{Power}=P(\text{reject }H_0\mid\text{specified alternative and model})=1-\beta$

A design with 80% power for a specified difference Δ has a 20% failure-to-reject probability for that alternative, under the planning assumptions. It does not have 80% power for every possible real effect, and power is not the probability that an observed significant result is correct.

Power depends on the target effect, variability, independent sample size, significance threshold, test direction, and analysis design. If everything else stays fixed, a smaller α usually reduces power. Repeated measures, clustering, unequal allocation, and missingness can change the calculation.

Worked planning approximation

For a balanced independent two-group mean comparison, a two-sided normal approximation gives:

$n_{per\ group}\approx2\left(\frac{z_{1-\alpha/2}+z_{1-\beta}}d\right)^2$

With α = 0.05, power = 0.80, and d = 0.5, the full-precision calculation gives n ≈ 62.7910, rounded up to 63 per group. Exact t-based planning can differ; this formula is an approximation for this design.

If d halves to 0.25, the approximate n quadruples. That is why deciding the smallest meaningful effect in advance is central to planning.

Do not use observed power to reinterpret the same p-value

Computing power from the observed effect in the same sample generally adds little to interpreting its p-value and can be misleading. For a completed study, report the estimate and uncertainty interval. For a future study, plan using defensible target effects and variability, with sensitivity analysis.

A nonsignificant result with a wide interval can leave both important benefit and harm plausible. A narrow interval inside predefined practical-equivalence bounds addresses a different question more directly.

Report a result in layers

Original-unit effect and its direction.
Appropriate uncertainty interval.
Standardized effect, if useful, with its definition.
Test result and inferential assumptions.
Practical context and design limitations.

Do not attach a confidence interval for a raw difference to d and label it an effect-size interval. These are different quantities and can need different methods.

Practice

Two groups have mean difference 4 and pooled SD 10. What is d? Does it prove practical importance or causality?

Show the explanation

d = 4/10 = 0.4. It describes an observed standardized mean difference. Practical importance depends on what four units mean, and causality requires design support. An uncertainty interval remains necessary for judging precision.

Continue learning

Read sample size determination, use the sample size calculator for its stated approximate designs, and review hypothesis testing to separate a p-value from effect magnitude.