Statistical Significance

What Is Statistical Significance?

Statistical significance describes the incompatibility of data with a specified null model, assessed against a prespecified decision threshold. It does not distinguish a real program effect from all alternative explanations.

A p-value is the probability, under a specified null model and its assumptions, of a test statistic at least as extreme as the one observed. A prespecified threshold such as 0.05 is a decision convention. The p-value is not the probability that the null hypothesis is true, that the result occurred by chance, or that a program caused the outcome. Examine effect size and uncertainty separately.

Why It Matters

Statistical testing can help assess uncertainty in quantitative evaluations when the model and design are appropriate. It supports interpretation alongside effect size, data quality, and the evaluation design. Causal attribution requires a justified identification strategy, not a significance threshold.

• Making attribution claims: interpret statistical uncertainty within a credible causal design and assess alternative explanations
• Scaling interventions: deciding whether to expand a program based on evaluation results that may reflect random variation
• Reporting to donors: providing defensible evidence of impact that meets methodological standards
• Avoiding false positives: prespecify analyses, address multiple testing, and report all relevant results rather than selecting significant findings

However, statistical significance alone is insufficient. A result can be statistically significant yet practically meaningless (tiny effect with large sample), or practically important yet not statistically significant (large effect with small sample). Practitioners must examine both statistical significance and effect size to fully interpret evaluation findings.

In Practice

Statistical significance appears primarily in quantitative impact evaluations and quasi-experimental designs. Common applications include:

Impact evaluations using randomized controlled trials (RCTs) or quasi-experimental designs calculate p-values for each outcome indicator to test whether treatment and control groups differ significantly. For example, a health program might find that vaccination rates are 15 percentage points higher in the treatment group (p=0.02), indicating incompatibility with the specified null model under its assumptions. It does not mean a 2% probability that chance explains the difference.

Survey analysis uses significance testing to determine whether observed differences across demographic groups (disaggregation) reflect real patterns or sampling variation. Interpretation must account for the sampling design, nonresponse, measurement error, and multiple comparisons.

Before-after comparisons can test a specified no-change model using the correct pairing, variance, and survey design. Even p=0.02 in an uncontrolled before-after study does not establish that the program caused the change; secular trends and other causes remain possible.

Best practice requires reporting both p-values and effect sizes (e.g., Cohen's d, odds ratios) alongside confidence intervals. A result showing p=0.049 should not be treated as meaningfully different from p=0.051 - the arbitrary 0.05 threshold creates a false binary. Instead, interpret the full statistical picture: effect magnitude, precision (confidence intervals), and practical relevance to program goals.

Method Source

ASA Statement on Statistical Significance and P-Values (2016), principles 1-6. This is methodological guidance, not a funding requirement.