Nonresponse Bias: How to Check Who Is Missing

What Is Nonresponse Bias?

Nonresponse bias is distortion in an estimate because people who do not respond differ from those who do in ways relevant to that estimate. A low response rate signals a reason to investigate; it does not measure the size or direction of bias. A high rate also does not guarantee that an estimate is unbiased.

The problem is specific to the question being answered. Missing mobile households may matter greatly for an estimate of service access but less for a characteristic that is similar across respondents and nonrespondents. Start with the intended estimate, the target population and the people actually represented. The wider bias question includes sampling, coverage, measurement and other errors as well.

Separate Missing People From Missing Answers

Keep refusal, inability to contact, uncertain eligibility and other final outcomes distinguishable. A respondent may provide sufficient information for the overall survey yet omit a sensitive question. That record can count toward one response measure while remaining missing for a particular estimate.

Calculate a Rate With a Defined Denominator

Choose a documented disposition and response-rate convention suited to the design. State how complete and partial interviews count, how eligibility is determined, and how unknown eligibility is handled. For a simple unweighted example with known eligibility, 120 responding units from 200 selected eligible units gives a 60% response rate. This is an illustrative calculation, not a complete formula for every survey.

Do not divide by everyone who clicked an open link and call it a probability-sample response rate. For a specific item, define the eligible people who should answer; exclude valid skips from that item's response-rate denominator. Report whether a rate is weighted or unweighted. Weights used to describe response may differ from final weights adjusted for nonresponse.

Look for Evidence Relevant to the Estimate

Begin with information available for both respondents and nonrespondents in the original sample, such as location, age group or program participation category, where lawful and sufficiently accurate. Compare response patterns across those groups. Distinguish an observed imbalance from a claim that the outcome itself is biased.

Useful checks include:

• Frame Comparisons: Compare the responding sample with known characteristics of the eligible selected sample and, where appropriate, the target population.
• External Benchmarks: Compare compatible measures against a credible source, checking population, time period and definitions first.
• Targeted Follow-Up: Seek information from a designed subsample of initial nonrespondents, accounting for its selection and remaining nonresponse.
• Sensitivity Checks: Show how conclusions change under stated plausible assumptions about missing outcomes.

Comparing early and late respondents can reveal a pattern, but late respondents are not automatically representative of people never reached. Agreement on observed characteristics also cannot prove agreement on unobserved outcomes. Keep these limits visible beside the results, including disaggregated findings.

Reduce the Problem Before Relying on Adjustments

During collection, examine contact success by relevant group. Review interview times, language access, accessibility and whether the questions create avoidable burden. Use the agreed contact protocol and consent protections. A targeted follow-up plan is more informative than replacing a hard-to-reach selected person with an easy-to-find volunteer.

Weighting may reduce bias when suitable auxiliary information relates to response and the outcome, the model is adequate and relevant groups remain represented. It cannot automatically recover an absent group or correct unmeasured differences. Very large adjustments may also reduce precision. Survey design and variance estimation need to reflect the weights used.

Imputation addresses missing values under assumptions; it does not make the values observed. Simple substitution of a mean can hide variation and understate uncertainty. Where item nonresponse affects an important finding, use an approach suitable for the data and analysis, with statistical support when needed, and disclose the assumptions and uncertainty.

A Hypothetical Service-Access Example

Suppose an evaluation samples 200 eligible households, equally divided between near and remote areas. Eighty near households and forty remote households respond. The overall response rate is 60%, but the group rates are 80% and 40%. Remote households are underrepresented among respondents.

Imagine reported access among respondents is 75% near and 50% remote. The unweighted responding-sample figure is about 66.7%. Reweighting the groups to their known equal population shares gives 62.5%. These hypothetical numbers demonstrate how an observed composition difference can affect an estimate; they do not establish that 62.5% is the true population rate. Nonrespondents within each area may still have different access.

Report What the Checks Establish

Keep a table of final contact outcomes, the response-rate definition, group comparisons, corrective actions and the limits of the assessment. Report important item-missingness patterns alongside each affected measure. Explain whether the conclusion changes under plausible missing-outcome assumptions.

Where the evidence does not settle the risk, narrow the claim. “Among responding households” may be more defensible than a population claim, although it changes the question being answered. Retain the original records and documented adjustments as part of data quality assurance.

Frequently Asked Questions

Does a Low Response Rate Prove Bias?

No. It raises concern, but the bias in an estimate depends on relevant differences between respondents and nonrespondents. Report the rate and assess those differences using the information available.

Can Weighting Remove All Nonresponse Bias?

No. Its usefulness depends on the auxiliary information, representation and assumptions. Residual differences may remain after weighting, particularly when they concern outcomes not explained by the adjustment variables.