At a Glance
Teams often pick a method first ("we need an impact evaluation") and then look for questions it can answer. This page runs the other way. It starts with a decision, writes the question that decision depends on, and then asks what evidence, timing and effort each question needs. The how to choose an evaluation methodology page compares the design families in detail; this one shows how to get from a question to the shortlist.
At a Glance
| Q1: Who Is Reached? | Q2: Why Do People Leave? | Q3: What Happens to Graduates? | Q4: Did It Change Employment? | |
|---|---|---|---|---|
| Decision it informs | Adjust outreach this quarter | Redesign the first weeks | Report results to the funder | Whether to expand |
| Evidence needed | Enrollment records, target profile | Attendance data, accounts from leavers and staff | Employment status after training | Outcomes for participants and a credible comparison |
| Design that can answer | Descriptive analysis of records | Process evaluation | Follow-up survey of graduates | Randomized, regression discontinuity or comparison-group design |
| Cannot show | Whether the program helped | How much leaving affects outcomes | What would have happened without the program | Why it worked, unless paired with qualitative work |
| Burden | Low | Medium | Medium | High |
| Ready before the decision? | Yes | Yes | Yes, for earlier cohorts | Not with current records |
Worked Case
Worked Case: A Youth Skills Program
This fictional example illustrates the reasoning. It is not a template for any particular study, and the figures are made up.
A provincial program offers a four-month vocational course, followed by job placement support, to young people aged 18 to 24 in six districts. About 2,400 have enrolled over two years. The provincial board meets in nine months to decide whether to expand to six more districts. Staff have noticed that many participants leave in the first weeks, and the funder asks for results.
The board's real question is whether to expand. That is too broad to evaluate directly, so the team breaks it into four questions, each tied to a use.
Question 1: Who Is Reached?
Use: Staff want to adjust outreach this quarter. The program targets out-of-school youth, with a goal of at least half women.
Evidence: Enrollment forms and the district lists the program was meant to reach.
Design: Descriptive analysis of the program's own records. Check completeness first: which fields are missing, and for whom. Compare enrollees with the target profile by district, sex and school status.
Limits: This describes who enrolled, not whether they benefited. If the forms record school status inconsistently, report that.
Burden: Low. About two to three weeks of analysis, with no new data collection.
Question 2: Why Do Participants Leave Early?
Use: Program managers want to redesign the first four weeks, when most leaving occurs.
Evidence: Attendance records show when people leave. Accounts from those who left, those who stayed and the trainers show why.
Design: A small process evaluation. Use attendance data to describe timing and groups. Interview a purposive set of leavers and stayers, chosen to vary by district, sex and age, with consent and a safe way to reach them. Look for explanations that appear in more than one source, and for accounts that do not fit. Agreement between sources raises confidence but does not confirm an explanation, because the sources can share biases.
Limits: The interviews generate explanations, not proportions. This design cannot say how much each reason accounts for, or whether fixing it would improve employment. Interviews with leavers may miss those who could not be reached.
Burden: Medium, mostly interviewer and analysis time. Results can reach the managers within a few months.
Question 3: What Happens to Graduates?
Use: The funder wants to know how many graduates are working six months after the course.
Evidence: Employment status, hours and earnings, collected by a follow-up survey of graduates who finished at least six months ago.
Design: A follow-up survey, with the list of graduates as the sampling frame. Plan the sample size and a tracking procedure for graduates who move. Decide in advance how to treat nonrespondents, and report the response rate.
Limits: This covers graduates only. People who left early, whom Question 2 studies, are outside it, so the results describe those who completed the course. If the funder asks about everyone who enrolled, include a sample of leavers or say clearly that the figures apply to graduates. This design shows what graduates report, not what the program caused. Employment could be high because the program selects people already likely to find work, or because the labor market improved. A figure for the local youth employment rate from a labor force survey gives context, but it is not a counterfactual for these participants, who differ in age, interest and location.
Burden: Medium. Costs depend on tracking and travel.
Question 4: Did the Program Change Employment?
Use: The board wants to know whether employment is higher than it would have been without the program. This is the question behind the expansion decision.
Evidence: Employment outcomes for participants and a comparison group that can credibly be argued to represent what would have happened without the program. This is the counterfactual; see counterfactual.
Designs to consider, with the assumption each one rests on:
| Option | What It Needs | Key Assumption to Examine |
|---|---|---|
| Randomized selection | More applicants than places, and an ethical case for using a lottery | Applicants stay in their assigned groups, the groups do not contaminate each other and few are lost to follow-up |
| Regression discontinuity | Admission decided by a score above a cutoff, applied consistently | People just above and below the cutoff are alike, nobody can manipulate their score, and the result applies only near the cutoff |
| Comparison group with difference-in-differences | Outcomes before and after for participants and a comparison group | Both groups would have followed similar trends without the program |
| Phased expansion | New districts enter one after another, with the order set in advance | Districts that start later are comparable to those that start earlier |
Feasibility check: Everyone who applied so far was admitted on a first-come basis. There was no lottery, no admission score with a cutoff and no waiting list. The program holds no employment records for young people who did not take part, and no records from before the course. Randomization is no longer possible for current participants, and a regression discontinuity design has no cutoff to use. A difference-in-differences design would need outcomes before and after for a comparison group, and none exist. The existing records cannot support a credible comparison for the people already enrolled. A comparison built from new data on young people outside the program would be possible, but it would rest on strong assumptions about how similar those young people are, and it could not report much within the nine months.
Verdict: The team cannot credibly answer Question 4 for the existing cohorts using existing records. It can, however, plan for the answer. If the expansion goes ahead, introduce it in phases, with the order of districts chosen by lottery or by a documented rule, and collect the same outcomes in districts that have not yet started. That builds a comparison into the rollout. With only six new districts, the district is the unit of comparison, so a lottery gives comparability in expectation only, chance imbalance is likely and precision will be limited. Spillover between neighboring districts is also possible. The team should cost the extra data collection and report these limits.
Fitting the Questions to the Constraints
| Question | Ready Before the Decision? | Main Constraint | Result |
|---|---|---|---|
| Q1: Who is reached? | Yes | Quality of enrollment records | Do now |
| Q2: Why do people leave? | Yes | Reaching leavers safely | Do now |
| Q3: What happens to graduates? | Yes, for earlier cohorts | Tracking and response | Start now with graduates of six months or more |
| Q4: Did it change employment? | Not with current records | No comparison group or earlier records | Stage it into the expansion |
For each question, check four things before you commit: the decision date, the evidence you can reach, the burden on participants and staff, and the cost. The evaluability assessment page gives a fuller checklist, and costing an evaluation shows how to price each task. Record the chosen questions and evidence in an evaluation matrix.
Using It
When the Question Is Larger Than the Design
Sometimes the question you need to answer is bigger than any design you can afford or finish in time. Three honest options remain:
- Narrow it. Replace "did the program raise employment?" with "how many graduates are working, and what do they say helped?" Then word the findings to match.
- Stage it. Answer the parts you can now, and set up the rest, as in the phased expansion above.
- Accept a weaker claim and label it. A theory-based approach such as contribution analysis can assess whether the evidence is consistent with the program contributing to the change, and how plausible the rival explanations are. It does not estimate how much of the change the program caused.
Whichever you choose, write the claim the evaluation will not support into the terms of reference, so the funder agrees before the work starts.
Checking an AI-Drafted Design
An AI assistant can draft a design from a question quickly. Treat the draft as a proposal to examine, not a recommendation. Give it the decision, the date, the data you have and your budget, then check:
- Does each design name what it cannot show? If a draft says a before-and-after comparison "measures impact", mark it.
- Does it assume a comparison group you do not have? Ask where the group comes from and what happens if it is not there.
- Do the dates work? Compare the follow-up time with the decision date.
- Are the numbers real? Sample sizes, costs and typical effects in the draft are suggestions. Recompute them from your own inputs, and ask for sources you can open.
- Who decides? The choice of design, the ethics and the final claim stay with the people who answer for the evaluation. See verifying AI outputs.
Common Mistakes
Starting from a method. "We will do a randomized trial" tells you nothing about whether it answers the decision.
Letting one question carry four. "Evaluate the program" bundles reach, process, outcomes and effect. Separate them, and each gets its own evidence and timing.
Reading outcomes as effects. Employment among graduates is an outcome. An effect needs a credible comparison.
Ignoring when results arrive. A strong design that reports after the decision informs the next decision, not this one.
Choosing the design to match the budget and then keeping the original question. If the budget changes the design, change the claim as well.
Hiding burden. Interviews, tracking and repeated surveys cost participants time. Count that as part of the feasibility check.