Impact Evaluation

When Is Impact Evaluation the Right Approach?

Impact evaluation is the right approach when you need to know whether a program caused observed changes in outcomes. Outcomes improving is not the same as the program causing them. That distinction is what impact evaluation is designed to establish. This is a high bar that requires substantial investment in design and data collection. Use it when:

• Scale decisions depend on evidence: governments or donors considering large-scale rollout need credible evidence that the program works before committing resources
• Program effectiveness is genuinely uncertain: the intervention has a plausible theory of change but has not been rigorously tested in this context
• Policy competition exists: comparing two alternative approaches requires a comparative design to determine which is more effective
• An agreed decision requires causal evidence: check the evaluation scope and deliverables in the applicable funding agreement
• The stakes are high: programs that affect large numbers of people or involve significant resources warrant the investment in rigorous evaluation

Impact evaluation is not appropriate when the program is still being developed (use formative evaluation first), when outcomes cannot be measured in the program timeline, when a counterfactual cannot be ethically or practically constructed, or when the evaluation question is about how outcomes occurred rather than whether they did (use contribution analysis or process tracing instead).

How Does Impact Evaluation Work?

All impact evaluations rest on one central idea: the counterfactual: what would have happened to program participants in the absence of the program. Since you cannot observe the same people both with and without the program, you construct a comparison group that approximates this counterfactual.

Step 1: Plan at the design stage

Plan prospectively where possible. Retrospective designs can be credible when assignment rules or existing data support a defensible counterfactual. Baseline requirements depend on the design.

Step 2: Define the evaluation question

State precisely what outcome you are trying to measure, for whom, over what time period, and at what geographic level. Vague questions produce inconclusive evaluations.

Step 3: Choose a design

The design choice depends on whether random assignment is feasible:

• Randomised Controlled Trial (RCT): participants are randomly assigned to treatment or control. Gold standard for internal validity but costly and often ethically difficult
• Quasi-experimental designs: when randomisation is not possible: difference-in-differences, propensity score matching, regression discontinuity, or interrupted time series. See quasi-experimental design for details, or how an RCT compares with a quasi-experimental design

Step 4: Establish baseline

Collect the pre-intervention data required by the selected design. A valid post-test randomized comparison can identify a treatment effect without a baseline; baseline measurements often improve precision and understanding. DiD requires suitable pre-period observations. Document balance and use a prespecified analysis rather than automatically adjusting every observed difference.

Step 5: Implement with evaluation integrity

Monitor for contamination (comparison group accessing program), attrition (losing study participants), and design fidelity (program delivered as intended). These threats to validity must be managed throughout implementation.

Step 6: Collect follow-up data and analyze

Collect midline and endline data at pre-specified intervals. Analyze using the appropriate statistical methods for the chosen design. Report the treatment effect size with confidence intervals, not just significance tests.

Step 7: Interpret and communicate findings

A statistically significant effect is not the same as a practically meaningful one. Report effect sizes in terms decision-makers understand (absolute changes, percentage changes, lives affected) alongside statistical significance.

Key Components

• Counterfactual: a credible comparison group that approximates what would have happened without the program
• Design-specific pre-intervention data: needed for some designs and often useful for precision, but not compulsory for every post-test randomized design
• Primary outcome indicator: one or two key outcomes the evaluation is powered to detect
• Sample size calculation: determines how many participants are needed to detect an effect of expected magnitude
• Pre-registration: document the design, hypotheses, and analysis plan before outcome analysis, using a suitable registry when applicable
• Follow-up data: midline and endline measurements at pre-specified intervals
• Analysis plan: pre-specified statistical methods to prevent data dredging

Best Practices

Commit to the counterfactual. The entire credibility of an impact evaluation depends on the quality of the comparison group. Random assignment is the gold standard; when it is not feasible, document carefully why and use the best available quasi-experimental design.

Match baseline needs to the design. No baseline does not automatically rule out impact evaluation. A valid post-test RCT compares randomized groups at follow-up. An uncontrolled before-after comparison, even with a baseline, cannot by itself rule out secular trends or other causes.

Power the study to detect realistic effects. Underpowered studies produce inconclusive results regardless of how well everything else is done. Work with a statistician to calculate minimum sample sizes based on expected effect sizes.

Use the same instruments across groups. Survey tools and questions must be identical between treatment and comparison groups to ensure comparability.

Pre-register the design. Pre-registration prevents selective reporting of positive findings and builds credibility with donors and policymakers. 3ie, AEA RCT Registry, and RIDIE are the main registries.

Common Mistakes

Starting without a defensible design. Planning late can remove design options and pre-intervention data. Assess the available assignment mechanism, records, and assumptions before deciding whether retrospective causal inference is credible.

Leaving mechanism and heterogeneity questions unplanned. An impact evaluation can estimate subgroup effects and examine mechanisms when the design, sample size, and measurements support them. Add process inquiry where it helps explain findings.

Inadequate attention to comparison group quality. Propensity score matching, difference-in-differences, and regression discontinuity all depend on assumptions that must be tested and reported. Presenting quasi-experimental results without discussing the plausibility of design assumptions is misleading.

Conflating statistical significance with program success. A statistically significant effect of negligible magnitude is not a program success. Report and interpret effect sizes.

Neglecting negative results. Null results are information. A well-conducted impact evaluation that finds no effect is valuable evidence. Suppress null results and you distort the evidence base.

Examples

These scenarios and effect estimates are illustrative rather than sourced empirical studies.

Agricultural livelihoods, East Africa. A food security program in Ethiopia used a quasi-experimental design with propensity score matching to evaluate impact on household dietary diversity and income. Baseline data was collected for 3,000 treatment households and 2,400 matched comparison households before program start. Midline and endline surveys tracked outcomes over five years. The illustrative estimate was a 0.8 standard deviation improvement in dietary diversity scores relative to matched comparisons. Attribution would require a justified identification strategy and assumptions about unmeasured confounding and changes over time; matching alone is insufficient. The effect was concentrated in female-headed households, prompting a design revision for the follow-on program.

Health, West Africa. A malaria prevention program in Nigeria used a cluster-randomised trial design, randomising 60 communities to treatment (free bednet distribution plus community health worker visits) or control (free bednets only). The evaluation found that adding community health worker visits produced a 23 percentage point increase in consistent bednet use relative to bednets alone, informing an assessment of whether the additional benefits justified component costs. The effect estimate alone does not establish cost-effectiveness or transferability to national scale-up.

Education, South Asia. A learning improvement program in Pakistan used a regression discontinuity design based on school-level test score rankings to evaluate impact on student achievement. Schools just below the eligibility threshold were compared to schools just above. The evaluation found a 0.4 standard deviation improvement in literacy scores among Grade 3 students in program schools, with larger effects for girls and rural schools.

Compared To

Relevant Indicators

52 donor-aligned indicators across USAID, DFID, World Bank, 3ie, USDA, and Global Fund. Key examples:

• Net attributable change in primary outcome between baseline and endline (treatment vs. comparison)
• Effect size (Cohen's d or percentage point difference) at program completion
• Proportion of evaluation hypotheses confirmed versus disconfirmed
• Fidelity score for program implementation as designed

Related Tools

• Evaluation Planner: structure your evaluation design and timeline from program start
• Indicator Library: find donor-aligned outcome indicators for your sector

Method Source

World Bank and IDB, Impact Evaluation in Practice, second edition (2016), chapters 4 and 15. This is methodological guidance, not a funding requirement.