Survey Design

When Should You Run a Survey?

Use survey design when you need to collect structured, comparable data across a large number of respondents to answer specific quantitative questions about a population. Surveys are the primary instrument for baselines, midlines, and endlines. They are also used for needs assessments, coverage monitoring, and performance surveys. Use them when you need data that is systematic, replicable, and statistically representative.

Surveys are not the right tool when you need to understand why something is happening (use focus group discussions or key informant interviews), when the population is too small for statistical analysis (use qualitative methods), or when the question requires narrative or interpretive answers.

How It Works

Step 1: Define the evaluation questions the survey must answer

Every survey item should trace directly to an evaluation or monitoring question. Items without a clear question-to-indicator link should be removed. Unfocused surveys produce data that no one uses.

Step 2: Draft the instrument

Write items using clear, simple language. Each item should measure one thing. Avoid double-barrelled questions ("Do you feel safe and supported?"), leading questions, and jargon. Use established, validated instruments wherever they exist (e.g., HDDS for dietary diversity, WDDS for women's dietary diversity, MDD-W for minimum dietary diversity).

Step 3: Design the question flow and skip logic

Organize items into logical sections. Use skip logic to route respondents past irrelevant sections. Begin with non-sensitive, rapport-building questions. Place sensitive items (income, violence) toward the end.

Step 4: Pilot the instrument

Test the draft with a small sample (15-30 respondents) from the same population type as the study. Identify misunderstood items, translation issues, and skip logic errors. Revise based on findings. Do not skip piloting - it is the single highest-return investment in data quality.

Step 5: Train enumerators

Enumerators must be trained on the instrument, interview protocols, consent procedures, and data entry. Run calibration exercises where pairs of enumerators interview the same respondent independently and compare results.

Step 6: Implement with quality controls

Use digital data collection (SurveyCTO, KoBoToolbox, ODK) to enforce skip logic, range checks, and required fields. Conduct field supervision with back-check surveys (re-interviewing a random 10% sample to verify enumerator data). Review daily data reports during data collection.

Key Components

• Coverage: which topics and indicators are included, and which are deliberately excluded
• Question types: Likert scales, multiple choice, open-ended, ranking, observation-based
• Response categories: exhaustive, mutually exclusive, and appropriate for the population's understanding
• Skip logic: routing that prevents irrelevant questions and reduces respondent burden
• Translation and back-translation: if conducted in a language other than English, translate forward, then independently back-translate to verify meaning
• Piloting protocol: plan for who, where, and how the instrument will be tested before deployment
• Data entry and validation rules: built-in range checks and required fields for digital data collection

Best Practices

Use validated instruments. Reinventing widely used instruments (dietary diversity, food security, WASH) introduces comparability problems and quality risks. Use established tools with documented validity and reliability where they exist.

Collect outcome data, not just output data. Many surveys track what was delivered (outputs) rather than what changed (outcomes). Outcome indicators require outcome questions.

Match baseline data to the design. Pre-intervention measurements support change analysis and can improve precision. A valid post-test randomized comparison can estimate an effect without baseline outcomes; before-after observations alone do not prove causation.

Match survey timing to measurement logic. Some outcomes need time to materialise. Collecting endline data 3 months after a 2-year program intervention may be too early to detect genuine change.

Keep instruments short. Respondent fatigue produces lower quality data in the second half of long surveys. Aim for under 45 minutes for household surveys. Every item cut improves data quality on the items that remain.

Common Mistakes

Over-designing the instrument. Adding items "just in case" produces surveys that are too long, tire respondents, and generate data that is never analyzed. Every item costs respondent time, enumerator time, and analysis effort.

Skipping the pilot. Pilots reveal translation problems, confusing items, and skip logic errors that are invisible on paper. Pilot scope should reflect population diversity, sensitive topics, routing, languages, and platform behavior; a small pilot does not establish a fixed percentage of problems detected.

Collecting without a justified use or ethical basis. Assess decision value, accountability obligations, potential harms, consent, referral arrangements, and respondent burden. Important adverse outcomes may still need documenting even when this program lacks a remedial budget. Plan how findings will be communicated or referred; do not suppress required harm or accountability evidence simply because the result may be negative.

Failing to standardise across enumerators. If different enumerators interpret and administer items differently, the resulting data is not comparable. Calibration training and back-check protocols address this.

Examples

These scenarios are illustrative rather than verified studies. Use the exact instrument, version, eligible population, recall period, and scoring guidance; comparability requires checking those details.

WASH baseline, East Africa. A donor-funded WASH program in Ethiopia used the WASH Conditions Assessment Tool as the basis for its baseline survey, adding 12 program-specific items on hygiene behavior. The 40-minute household survey was piloted in two villages outside the program area before deployment. Calibration exercises between enumerator pairs identified a misunderstood definition of "improved latrine" that was corrected before field data collection. The final survey was administered to 1,800 households across three districts.

Food security survey, West Africa. A donor-funded program in Mali used the Household Food Insecurity Access Scale (HFIAS) and the Household Dietary Diversity Score (HDDS) as the core of its monitoring survey. The team planned comparisons with the program baseline and a relevant reference dataset, conditional on matching instrument versions, administration, eligible populations, recall periods, and scoring. Instrument names alone do not establish comparability. Local language translation used forward-translation by bilingual program staff followed by independent back-translation by a university linguist.

Compared To