Introduction to Hypothesis Testing

Test a specific claim about a population

A packaging line targets a mean fill of 500 g. A sample of 16 independently selected packs has mean 498 g and sample SD 4 g. Is this evidence that the population mean differs from the target?

A hypothesis test asks how compatible the observed result is with a specified null model. It does not calculate the probability that the claim is true or decide whether a difference is important enough to act on.

Define the question before looking at the result

For this example:

• Population: packs from the defined production period.
• Parameter: population mean fill μ.
• Null hypothesis: H₀: μ = 500 g.
• Alternative: H₁: μ ≠ 500 g.
• Significance level: α = 0.05, chosen before analyzing the sample.

The alternative is two-sided: both underfilling and overfilling matter. A one-sided alternative is appropriate only when its direction is justified in advance and effects in the other direction would not support that claim. Do not choose a direction after seeing the data.

A null need not mean “no effect.” It can be any specified parameter value, such as a mean of 500 g or a success proportion of 0.20.

Choose a statistic and check the design

Because population SD is unknown, use a one-sample t statistic:

$t=\frac{\bar{x}-\mu_0}{s/\sqrt n}$

For exact small-sample t inference, observations should be independent and normally distributed. In practice, inspect the sample’s shape and extreme values and consider robustness. A large sample can help with some departures, but cannot fix systematic sampling bias or dependence.

If all 16 packs came from one unusually quiet hour, generalizing to the entire production period needs caution. Statistical arithmetic is the last step in a chain of design assumptions.

A complete calculation

1. Standard error: SE = 4/√16 = 1 g.
2. Statistic: t = (498 − 500)/1 = −2.
3. Degrees of freedom: df = 16 − 1 = 15.
4. Two-sided p-value: P(|T₁₅| ≥ 2) ≈ 0.0639.
5. Decision at α = 0.05: fail to reject H₀, because 0.0639 > 0.05.

The t-table gives a two-sided 5% cutoff of about ±2.1314 for df = 15. Since |−2| < 2.1314, the critical-value approach gives the same decision.

Interpret the p-value accurately

The p-value 0.0639 means: if the null model and its assumptions hold, the probability of a t statistic at least as extreme as ±2 is about 6.39%.

It does not mean:

• H₀ has a 6.39% probability of being true.
• There is a 6.39% probability that chance caused the result.
• The process is acceptable because the test was not significant.
• A larger p-value would prove the absence of a meaningful difference.

The American Statistical Association’s statement emphasizes that significance alone does not measure effect size or justify a scientific conclusion. Keep the estimate, uncertainty, design, and context together.

Report the interval and practical meaning

The matching 95% confidence interval is:

$498\pm2.1314(1)=[495.8686,500.1314]\text{ g}$

A useful report is: “The sample mean was 498 g, 2 g below target. The 95% t interval for the population mean was approximately 495.87–500.13 g; the two-sided test gave t(15) = −2.00, p = 0.0639.”

The interval includes the target but also includes values more than 4 g below it. If underfilling of that size matters, the result is inconclusive for operational purposes. “Not significant” is not the same as “close enough.” Practical equivalence needs predefined tolerances and an appropriate equivalence analysis.

Understand the two kinds of test error

Reality and decision	Name
Reject a true null	Type I error
Fail to reject a false null	Type II error

For a correctly calibrated test, α controls the long-run Type I error rate under the null. It is not the probability that this particular rejection is wrong. Type II error, β, depends on a specified alternative effect, sample size, variability, and the procedure; power is 1 − β.

A smaller α generally requires stronger evidence to reject and can reduce power if everything else stays fixed. Plan the trade-off based on the costs of errors rather than changing the cutoff after seeing p.

Avoid testing until something becomes significant

Testing many outcomes, repeatedly checking results, or changing exclusions after seeing p changes the behavior of the analysis. Twenty independent tests at α = 0.05 have probability 1 − 0.95²⁰ ≈ 64.2% of at least one rejection when all nulls are true.

Predefine primary outcomes and the analysis plan. Use suitable multiplicity adjustments or sequential designs where needed. Report exploratory analyses as exploratory; do not present a discovered pattern as a preplanned test.

Practice

1. A two-sided test gives p = 0.04. What decisions follow at α = 0.05 and α = 0.01?

Show the explanation

Reject at 0.05, fail to reject at 0.01. The underlying data and p-value do not change. Neither decision tells you the probability H₀ is true or whether the effect is large enough to matter.

2. A study reports p = 0.30 and a wide interval spanning large benefits and harms. Can it claim “no effect”?

Show the explanation

No. The estimate is imprecise and the data do not distinguish several substantively different effects. A narrow interval inside predefined equivalence bounds would address a different question more directly.

Continue learning

Read t-tests for one-sample, independent, and paired designs; chi-square tests for categorical counts; and effect size and power for planning and interpretation.