At a Glance
A follow-up survey rarely reaches everyone. The usual answer is a response or retention rate, but a rate says how many are missing, not whether their outcomes differ from the rest. This page helps you decide whether a result still stands once you allow for the people you could not measure. Nonresponse bias explains how to detect and reduce the problem; this page shows how to test a conclusion against it. All numbers are made up.
You need three things before you start: who was selected or assigned (the full list, not only those measured), the outcome and group for each person, and the retention by group. If you have a baseline, keep it; it is the best information on who left.
At a Glance
| Approach | What It Assumes | What You Get | What It Cannot Do |
|---|---|---|---|
| Observed only | The missing people resemble the observed ones, in every group | A single estimate | Show how much the answer depends on that assumption |
| No-assumption bounds | Only that outcomes lie in a known range | The widest range the data allow | Narrow the range; it is often too wide to decide anything |
| Tipping point | Nothing fixed; you vary the missing outcomes | How different the missing would have to be to change the conclusion | Say how likely that difference is |
| Trimming bounds | Random assignment, and the program moves response in one direction only | A narrower range for people who would respond either way | Cover people who respond only under one condition |
| Weighting by measured traits | Within each measured group, missing people resemble observed ones | A corrected point estimate | Correct for traits you did not measure |
Worked Case
Worked Case: A Job Program With Unequal Loss
A fictional program offers job placement support. Of 600 young people, 300 were assigned to the program and 300 to a comparison group by lottery. Random assignment gives groups that were alike at the start. It does nothing about who answers the endline survey. Whether a person is found and agrees to answer is a separate selection process, and the program may change it.
| Program | Comparison | |
|---|---|---|
| Assigned | 300 | 300 |
| Interviewed at endline | 240 | 270 |
| Retention | 80% | 90% |
| Employed, among those interviewed | 156 | 135 |
| Share employed, among those interviewed | 65.0% | 50.0% |
Overall retention is 85%, so overall attrition is 15%. That figure hides the gap: 20% of the program group were lost, against 10% of the comparison group. Everything below treats the 600 as the whole group of interest, so no sampling error appears. If they were a sample, add the interval methods in reading a survey estimate.
Step 1: The Observed-Only Result
Among those interviewed, 65.0% of the program group and 50.0% of the comparison group are employed, a difference of 15.0 points. This is the right answer to one question: how do the people we reached compare? It answers the program question only if the 60 missing program participants and 30 missing comparison participants would have been employed at the same rates as their groups' interviewed members. That is an assumption about outcomes nobody saw. In the missing-data literature, this is the missing at random condition applied within each group: missingness may depend on what was observed, but not on the unseen outcome itself (Rubin 1976; National Research Council 2010). It cannot generally be verified from the observed data (National Research Council 2010).
Step 2: Bounds With No Assumptions
Each missing person is either employed or not. Count all of them as not employed for the lowest value, and all as employed for the highest.
| Group | Lowest Share Employed | Highest Share Employed |
|---|---|---|
| Program | 156 / 300 = 52.0% | (156 + 60) / 300 = 72.0% |
| Comparison | 135 / 300 = 45.0% | (135 + 30) / 300 = 55.0% |
The program effect is lowest when the program group is at its lowest and the comparison group at its highest: 52.0 - 55.0 = -3.0 points. It is highest at 72.0 - 45.0 = 27.0 points. Without assumptions, the data cannot rule out a zero or slightly negative effect, nor a very large one. This is the logic of Manski's bounds (1989). The range is wide because 15% of the people (90 of 600) are missing, and for a yes or no outcome the missing share translates directly into uncertainty.
Step 3: A Tipping-Point Check
Instead of the extremes, vary the share employed among the missing people in each group, and read the effect across the full set of assumptions.
| Employed Among Missing Program Participants | Missing Comparison: 50% | 65% | 70% | 75% |
|---|---|---|---|---|
| 0% | 2.0 | 0.5 | 0.0 | -0.5 |
| 25% | 7.0 | 5.5 | 5.0 | 4.5 |
| 50% | 12.0 | 10.5 | 10.0 | 9.5 |
| 65% | 15.0 | 13.5 | 13.0 | 12.5 |
| 100% | 22.0 | 20.5 | 20.0 | 19.5 |
Each cell is the program effect in points for all 600 assigned. The cell for 65% and 50% is the observed-only result of 15.0, because it fills in each group's missing people at that group's own interviewed rate.
The effect reaches zero or below only in the top right. With none of the 60 missing program participants employed, it needs about 70% or more of the 30 missing comparison participants employed; with all 30 comparison participants employed, it needs 15% or fewer of the program's missing employed. Both are far from the 65% and 50% observed. If the missing comparison participants were employed at the observed comparison rate of 50%, the effect stays at +2.0 points or more even if no missing program participant has a job. Whether the tipping scenario is plausible is a judgment for people who know the setting. For example, if comparison-group youth migrated for work and could not be found, the tip becomes more believable than it first looks.
Step 4: Trimming Bounds
Lee (2009) narrows the range by adding two assumptions: assignment was random, and the program affects response in one direction only. Lee states the result for a continuous outcome and cites a separate implementation for yes or no outcomes; for a yes or no outcome like this one, the same logic works by counts, as below. Here the program group was retained less (80% against 90%), so we assume the program made nobody more likely to answer. Then the comparison group contains some people who would have been lost had they been in the program group. The share of such people among interviewed comparison participants is (0.90 - 0.80) / 0.90 = 11.1%, which is 30 of the 270.
We do not know which 30 they are, so trim the comparison group two ways:
| Trim | Comparison Employed | Effect |
|---|---|---|
| Remove the 30 who are not employed (comparison looks best) | 135 / 240 = 56.25% | 65.0 - 56.25 = 8.75 points |
| Remove 30 who are employed (comparison looks worst) | 105 / 240 = 43.75% | 65.0 - 43.75 = 21.25 points |
Under these assumptions the effect for people who would answer under either assignment lies between 8.75 and 21.25 points. The range is much narrower than -3.0 to 27.0, but it carries two costs. It describes only the people who would respond either way, not everyone assigned. And it is only as good as the one-direction assumption: if the program made some people easier to reach and others harder, the bounds do not apply. The method also needs the sample's own uncertainty added when the data are a sample.
Step 5: Adjusting for What Was Measured
The baseline records whether each person was employed before the program. Response differs by that trait:
| Program: Baseline Employed | Program: Not Employed | Comparison: Baseline Employed | Comparison: Not Employed | |
|---|---|---|---|---|
| Assigned | 100 | 200 | 100 | 200 |
| Interviewed | 90 | 150 | 95 | 175 |
| Employed at endline | 81 | 75 | 85 | 50 |
Within each group, weight the interviewed people back to the assigned counts. Program: (100 x 81/90 + 200 x 75/150) / 300 = 190 / 300 = 63.3%. Comparison: (100 x 85/95 + 200 x 50/175) / 300 = 146.6 / 300 = 48.9%. The adjusted effect is 14.5 points, slightly below 15.0.
This corrects only for baseline employment. It assumes that, among people with the same baseline status in the same group, those missing resemble those interviewed. That cannot be tested, and traits you did not measure, such as health or migration, stay uncorrected. See sampling weights for how nonresponse adjustments are built and what they do to precision.
What the Result Supports
| Method | Effect, Points | Assumption Carried |
|---|---|---|
| Observed only | 15.0 | Missing resemble observed |
| Weighted by baseline employment | 14.5 | Missing resemble observed within baseline groups |
| Trimming bounds | 8.75 to 21.25 | Random assignment; response moves one way |
| No assumptions | -3.0 to 27.0 | None |
A defensible summary: "Among interviewed participants, 65% of the program group and 50% of the comparison group were employed. Retention differed (80% and 90%). Adjusting for baseline employment gives 14.5 points. Allowing for the missing under a one-direction assumption gives 8.75 to 21.25 points for those who would respond either way. With no assumptions, a zero effect cannot be excluded." That statement does not claim the program worked for everyone assigned. It reports how the answer moves with the assumption. Differences in who is missing also change the question: see panel vs repeated cross-section for attrition planning, and counterfactual for why assignment matters.
Checking and Reporting
Check an AI-Run Sensitivity Analysis
AI tools can run these calculations quickly and describe them confidently. The check is to rebuild the table yourself and test the assumptions the tool used, not to ask it to review itself.
A fictional AI draft might say: "Attrition was 15%, within acceptable limits, so the 15-point difference is unbiased." Checking it:
Recompute retention by group from the counts: 240 / 300 = 80% and 270 / 300 = 90%. The draft's 15% is the overall figure and hides a 10-point gap.
Rebuild the bounds from the table with a spreadsheet or a short script. A brute-force check loops over every possible number of employed people among the 60 and 30 missing (61 x 31 combinations) and records the smallest and largest effect. It should return -3.0 and 27.0.
Ask for the assumption behind each figure, and mark each as testable or not. "Unbiased" should become "unbiased if missing outcomes resemble observed ones," which is not testable.
Check the claimed standard. If the draft cites an attrition threshold, ask for the source and read the passage. A threshold from one field or design does not transfer by default.
Record the model name and version, the date, and what you changed. The result depends on both.
The tool can speed up the arithmetic. Choosing the assumptions, judging which tipping scenarios are believable and deciding what to claim remain with you.
What to Put in the Report
The numbers selected, assigned and interviewed, by group, with retention for each.
The comparison of those retained and those lost, on baseline traits and baseline outcomes, by group.
The observed estimate and at least one sensitivity result, with the assumption written beside each.
The tipping scenario, in words a reader can judge.
The population the result describes, for example "people who would respond under either assignment."
Anything not covered, such as traits not measured and sampling uncertainty.
Common Mistakes
Reporting the overall attrition rate and stopping. The overall rate hides differences between groups, and differential loss threatens a comparison more than the total does.
Treating random assignment as protection against missing outcomes. Assignment makes the groups alike at the start. Who responds is a separate selection process.
Treating an adjustment as a repair. Weighting or filling in values works only to the extent that the assumption holds. Report it as a scenario.
Presenting bounds as confidence intervals. Bounds reflect uncertainty from the missing outcomes. They do not include sampling error, which needs its own interval.
Replacing missing people with whoever is easy to reach. A substitute is a different person. Keep replacements out of the main estimate or label them.
Using one method and dropping the others. The point of a sensitivity analysis is the range of answers. Show at least the observed result and one alternative.