At a Glance
Design an M&E questionnaire from the analysis plan backward. Decide what you will report first, then write only the questions that produce those numbers. Most weak questionnaires fail the same way: someone drafts questions that sound important, then tries to work out afterwards what can be analyzed from the answers.
The Seven Design Steps
Build a questionnaire in this order. Each step depends on the ones before it. Jumping ahead is the root cause of most questionnaire failures.
| Step | What happens |
|---|---|
| 1. Define the analysis plan | Specify the tables and comparisons the questionnaire must produce; each indicator traced back to a planned analytical output |
| 2. Map questions to indicators | One or more questions per indicator, with response options designed for the planned analysis |
| 3. Draft questions in plain language | Simple, direct, unambiguous wording; one concept per question; appropriate response scales |
| 4. Structure and sequence | Logical grouping, sensitive topics late, skip logic planned |
| 5. Translate and back-translate | Independent translator, comparison to English, discrepancy resolution |
| 6. Cognitive interview (5-10 respondents) | Respondent-level comprehension test, paraphrasing, surfacing ambiguity |
| 7. Field pilot (15 to 30 respondents, 30 to 50 if multi-language or multi-site) | Full protocol test including enumeration, timing, logistics |
Only after step 7 should a questionnaire go to full fieldwork. Skipping steps 5-7 is how preventable errors reach data collection and become permanent.
For the broader concept of survey design, see survey design.
Design from the Analysis Plan Backward
The single largest questionnaire design mistake is starting with the questionnaire and hoping analysis works out. Reverse this.
Step 1 is to produce a sketch of the analytical outputs the program will need at baseline, midline, endline, and donor reporting cycles. Each output table names:
- The indicator being reported
- Disaggregations (sex, age, location, vulnerability status, etc.)
- The comparison (baseline vs endline, program group vs comparison group, urban vs rural)
- The expected result shape (percentage, mean, count, scale score)
Once the analytical table is drafted, work backward: what question, asked in what form, with what response options, would produce exactly this cell in the table? Every question should trace to a specific analytical cell. Questions that do not trace back are candidates to cut.
This backward-design discipline does two things. It reduces questionnaire length by eliminating "it seemed relevant" questions that were never going to be analyzed. And it exposes indicator gaps at design time: if the analytical table has a row but no question produces that data, the gap is visible before fieldwork, not after.
Start With the Decision
The first question is not "what should we ask?" It is "what will we decide with the answers?" List the decisions the data will inform, then ask what single question would let someone make each decision well. Only that question belongs in the questionnaire.
Every question costs interview time, entry, cleaning and analysis. A simple test: beside each draft question, write the decision it informs, who makes that decision, and by when. A question that cannot fill all three is a candidate to cut.
| Decision | Question that informs it |
|---|---|
| Whether to extend the nutrition sessions to a second district | Since the sessions, did you change how you feed your child? What changed? |
| Whether to retrain the facilitators | What, if anything, was hard to understand in the sessions? |
Also check whether the data already exists. Household size or facility records may already sit in a registration database, and asking again wastes respondents' time.
Writing Questions
Question Wording Rules
Once you know what questions to ask, write them clearly. Eight rules produce cleaner data.
One concept per question. "Have you attended a training on hygiene and are you practicing what you learned?" is two questions. Split into: "In the past 12 months, did you attend a hygiene training?" and "[if yes] In the past 7 days, did you practice handwashing at the recommended critical times?"
Plain language, no jargon. "How frequently do you engage in income-generating activities?" fails if respondents are not sure what counts. "In the past 7 days, did you do any work for payment or goods?" is concrete.
Specific time frames. "Do you regularly attend community meetings?" is vague. "In the past 3 months, how many community meetings have you attended?" is measurable.
Avoid double negatives. "Do you disagree that the program should not expand?" loses half the respondents. Rewrite positively: "Should the program expand?"
Avoid leading or loaded wording. "How satisfied are you with the valuable training you received?" presumes the answer. "How would you rate the training you received?" is neutral.
Concrete units. "How much land do you farm?" is ambiguous (hectares, acres, local unit?). "How much land do you farm, in hectares or acres (please specify)?" is collectible.
Response scales with a neutral midpoint when measuring attitudes. A neutral midpoint (strongly disagree, disagree, neutral, agree, strongly agree) suits most attitude items. Evidence on 4-point forced-choice scales is mixed, so test the midpoint in the pretest, especially where respondents tend to agree with whatever is asked.
No implicit assumptions. "How many children do you have in school?" assumes the respondent has children and that the children are of school age. Lead with a screening question, or structure as "Do you have children of school age? [If yes] How many are currently enrolled?"
Choosing Question Types
Each question type suits some kinds of data and not others.
| Type | Best for, and its limit |
|---|---|
| Single-select | Categories, demographics. Easy to analyze, but forces a choice |
| Multi-select | Behaviors, lists of knowledge. Captures several answers, but order effects |
| Likert scale | Attitudes, satisfaction. Shows intensity, but acquiescence bias |
| Numerical | Counts, ages, quantities. Precise, but recall errors |
| Open-ended | Context, explanations. Respondent's own words, but hard to analyze at scale |
| Skip logic | Filtering out irrelevant questions. Lowers burden, but routing errors |
A Likert scale asks how far respondents agree on a ladder of options. Acquiescence bias is the tendency to pick the agreeable end whatever the true view. Where you can, ask about behavior instead of opinion: "In the past month, how many nutrition sessions did you attend?" is harder to answer falsely than "How satisfied are you with the sessions?"
Open-ended questions need thematic coding, so use one or two per survey at most.
Response Options and Scales
Response options carry as much weight as question wording. Three guidelines.
Exhaustive and mutually exclusive categories. Every possible answer fits in one and only one category. Add "other (specify)" where uncertainty exists. Do not force respondents into the nearest approximation.
Match the scale to the analysis. Ordinal scales (ranked categories, such as poor, fair, good) support different analyses than interval or ratio scales (numbers with equal steps, such as age or kilograms). Select the analysis for the measurement level and for the exact figure you want to report (for example, a percentage of households). Categorical and ordinal responses can support suitable regression models; treating an ordinal item as interval requires justification.
Numbers over labels for quantities. "How old are you?" with response options 18 to 29, 30 to 44, 45 to 59, 60 or more loses precision, and binning can happen at analysis but unbinning cannot. Collect exact values unless eligibility, the instrument standard or disclosure risk calls for bands.
Structuring the Instrument
Questionnaire Structure and Flow
A well-structured questionnaire opens with low-risk questions, groups related content, and places sensitive items after rapport is established.
Opening (about 10 to 20% of length, a rule of thumb): Introduction, consent, eligibility screening, low-risk contextual questions. Warm-up section that establishes the interview rhythm. Put full demographics last, unless you need them earlier for routing.
Core content blocks (about 60 to 70%): Grouped by theme, with smooth transitions between blocks. Each block should open with a transition statement ("Now I'd like to ask about your household's water access...").
Sensitive content (about 10 to 15%, placed late): Income, health status, GBV, legal status, illegal activities. Place after rapport is built and respondent is comfortable. Informed consent should have flagged that any question can be skipped.
Closing (about 5 to 10%): Thank-you, any final open-ended questions, contact for follow-up if needed, instructions for continuing interaction.
Within each block, move from general to specific, closed-ended to open-ended, and simple to complex. The opposite sequence causes respondent fatigue and dropout.
Skip Logic and Routing
Skip logic (also called routing) moves respondents through relevant questions while skipping irrelevant ones. Poor skip logic is a major source of missing data and enumerator confusion.
Design rule: every skip should be visually obvious on the questionnaire, or programmatically enforced in digital instruments. Ambiguous skips ("If no, go to section C" without specifying where section C starts) produce inconsistent enumeration.
Testing skip logic is part of pretesting. Cognitive interviews surface whether respondents understand the transition; field pilot tests whether the full logic executes correctly in practice. Digital tools enforce only the logic actually programmed, which may be wrong. Test every eligibility and skip path with ordinary, missing, refusal, and boundary cases; paper also needs explicit routing and trained enumerators.
Screen out respondents early when appropriate. If the questionnaire is about employed adults, ask about employment status first and terminate or redirect ineligible respondents. Dragging ineligible respondents through 30 minutes of irrelevant questions wastes field time and produces unusable data.
Pretesting & Translation
Cognitive Interviewing
Cognitive interviewing is the question-level pretest. 5-10 respondents from the target population work through the draft questionnaire aloud while an interviewer probes comprehension and answer-formation.
Standard cognitive-interviewing probes:
- "In your own words, what is this question asking?"
- "How did you arrive at your answer?"
- "Are there any words or phrases in the question that were unclear?"
- "Would different people interpret this question differently?"
What it surfaces: ambiguous wording, cultural mismatch, answer-formation difficulty, question sequencing problems. What it does not surface: workflow problems (enumerator handling, timing, logistics). That is what the field pilot is for.
Run cognitive interviews in the target language with target-population speakers. English-language cognitive interviewing of a Swahili questionnaire misses most translation-related comprehension errors.
Translation and Back-Translation
Translation is where English-language questionnaire design meets local reality, and where many errors originate.
Process:
- Draft in English
- Translate to target language(s) by a qualified translator
- Independent back-translation from target language to English
- Compare original English to back-translated English; flag discrepancies
- Resolve discrepancies (original author + translator + back-translator)
- Cognitive interview in target language with target-population speakers
Common translation errors: equivalent words with different connotations; culturally-specific concepts with no direct equivalent (household composition, kinship terms, privacy norms); idiomatic expressions that do not translate literally; technical terms that require coining or glossing in target language.
Back-translation alone does not catch all of these; cognitive interviewing in the target language is what surfaces concept-level mismatches. Budget time and cost for both.
Field Pilot
The field pilot is the full-protocol test: 15 to 30 respondents for a single-language, single-site survey and 30 to 50 for a multi-language or multi-site one, drawn from the target population, using the real enumerators, real logistics, real timing, and real digital (or paper) tools. It is not cognitive interviewing; it is workflow testing.
What the pilot surfaces:
- Total questionnaire time in practice (often longer than estimated)
- Enumerator consistency in handling skip logic, probing, and recording
- Instrument-level problems (digital form bugs, printing errors)
- Respondent fatigue points (where responses start shortening)
- Workflow bottlenecks (enumerator-supervisor review, data sync, quality checks)
Modifications after pilot: pilots often produce several questionnaire revisions, often 5 to 15 as a rule of thumb (wording, ordering, skip logic). Data from the pilot is usually not included in the main dataset unless the final instrument is identical.
Plan 1-2 weeks between pilot completion and full fieldwork start, enough time for revision, instrument re-compilation (if digital), and enumerator re-training on final wording.
Mistakes & Checklist
Sector Examples
The scenarios below are illustrative. Their figures, costs, and review outcomes are teaching assumptions, not reported studies or evidence of donor approval.
Health: Perinatal Care Knowledge Questionnaire, East Africa
A maternal health program designed a 28-minute questionnaire measuring perinatal care knowledge and practice among pregnant women. Analysis plan required disaggregation by age group (15-19, 20-24, 25+) and by geography (urban/rural). Cognitive interviewing with 8 respondents surfaced one critical finding: the term "perinatal" had no equivalent in the local language and the initial translation used a word that meant "after birth" only. The team revised to separate questions on pregnancy, delivery, and postnatal periods, gaining measurement precision at 3 minutes of additional length. Field pilot with 30 respondents showed that 18% of respondents gave shorter answers in the last block; the team moved two low-priority questions from that block to a follow-up visit.
Education: School Enrollment and Attendance Survey, South Asia
An education program's baseline survey asked parents about child school enrollment and attendance. Initial draft used "attends school" as binary. Cognitive interviewing showed respondents interpreted this differently: some counted any enrollment, some counted present-in-last-30-days, some counted regularly attending. The team revised to three separate questions (ever enrolled, currently enrolled, attended in last 7 days) allowing the analysis to distinguish the three outcome concepts. Back-translation caught a mistranslation of "regular attendance" that had produced a word meaning "daily without exception" rather than "most days."
WASH: Household Water Practices Survey, West Africa
A WASH program designed a household water and hygiene questionnaire. Pilot in 30 households revealed that the "improved water source" category definitions (JMP ladders) did not map cleanly onto local source types: "unprotected wells" and "boreholes with surface contamination risk" were being sorted inconsistently by enumerators. The team added a locally adapted checklist using JMP source-technology definitions. An unprotected well remains unimproved; a borehole is improved technology, while safely managed status also needs accessibility, availability, and quality evidence. Adaptation must preserve those distinctions. Total pilot time was 38 minutes, long for a household survey of this kind; the team moved three low-priority modules (modules on desired hygiene behaviors) to a separate follow-up study rather than cutting them.
Food Security: Seasonal Pastoralist Survey, Sahel
A food security program designing a survey for pastoralist communities found cognitive interviewing indispensable. Initial questions about "months of adequate food provisioning" (the MAHFP indicator structure) did not align with pastoralist seasonal patterns (wet/dry/transhumance rather than calendar months). The team revised to use seasonal markers (pre-migration, migration, post-migration, and wet season) that respondents could reliably identify, using seasonal prompts to recover the required month-by-month answers. If months cannot be reconstructed reliably, label the result an adapted seasonal measure rather than standard MAHFP. Back-translation caught the original mistranslation of "food gap" as "hunger death" which had produced alarming non-response rates in the pilot.
Common Mistakes
Mistake 1: Starting with the questionnaire, not the analysis plan. Questions that cannot trace to an analytical output usually produce unusable data. Build backward from planned tables, not forward from "important topics."
Mistake 2: Skipping cognitive interviewing. A questionnaire that has never been pretested with target respondents will have question-level comprehension errors you cannot see from the desk. Budget 1-2 days of cognitive interviewing before any pilot.
Mistake 3: Pilot-free fieldwork. Cognitive interviewing tests questions; piloting tests workflow. Both are needed. Skipping pilot means enumerator consistency, timing, and logistics problems arrive in full fieldwork.
Mistake 4: Double-barreled questions. One question, one concept. "Have you attended training AND are you practicing?" is two questions pretending to be one.
Mistake 5: Leading or loaded wording. Questions that presume the answer ("how much did you enjoy...") produce biased data. Neutral phrasing is harder but necessary.
Mistake 6: Binning quantities at collection. Choose numeric values or categories based on the required analysis, respondent burden, instrument standard, and disclosure risk, as set out under Response Options and Scales.
Mistake 7: Placing sensitive questions early. Income, GBV, legal status, health status should come after rapport is built. Early placement produces refusals and non-response.
Mistake 8: Translation without back-translation. Forward-only translation misses systematic errors. Back-translation + cognitive interviewing in the target language are the two controls.
Mistake 9: Ignoring skip logic testing. Skip logic in cognitive interviewing and in the field pilot both need explicit testing. Ambiguous skips produce missing data that cannot be recovered.
Mistake 10: No informed consent script aligned to the questionnaire. Consent should match what the questionnaire actually asks, including sensitive-question warnings and skip-option notices. See evaluation ethics checklist for consent-script requirements.
Pre-Fieldwork Questionnaire Checklist
Run through this before training enumerators.
Design:
- Analysis plan drafted first, with indicator-to-question mapping
- Every question traceable to a specific analytical output
- Questionnaire length within the target window for method (30-45 min in-person)
- No double-barreled, leading, or ambiguous questions
- Response options exhaustive and mutually exclusive
Translation:
- Translated to all target languages by qualified translators
- Back-translated independently and discrepancies resolved
- Target-language cognitive interview conducted
Pretesting:
- Cognitive interviewing complete with 5-10 target-population respondents per language
- Field pilot conducted (15 to 30 respondents, 30 to 50 if multi-language or multi-site) using full protocol
- Revision incorporated and instrument finalized
Protocol:
- Skip logic tested in both cognitive interviews and pilot
- Sensitive questions placed late with consent-script alignment
- Enumerator training covers interpretive edge cases, not just the instrument
- Inter-rater agreement plan specified
Logistics:
- Paper printing or digital instrument deployment tested
- Supervisor review workflow defined
- Data sync or collection protocol confirmed
- Ethics approval reflects the final instrument version
For related topics, see paper vs digital data collection, evaluation ethics checklist, the 5 data quality dimensions, and surveys vs interviews vs focus groups. For current document workspaces, see the Playbooks catalog.