Difference
The Difference Is What You Need to Learn
A questionnaire pretest examines whether people can understand questions and produce the information intended. A field pilot rehearses the wider survey process under conditions resembling the planned collection. The labels overlap: a pilot can include question pretesting, and pretesting can happen in the field. State the purpose and methods instead of relying on the label.
A completed interview is not proof that a question worked. A respondent may answer confidently while interpreting a reference period differently from the analyst. Equally, clear wording does not show that teams can reach selected households, obtain consent, synchronize records and protect files. A useful survey design addresses both kinds of risk.
| Decision | Questionnaire Pretest | Field Pilot |
|---|---|---|
| Main question | Do respondents understand and answer as intended? | Can the planned collection process work? |
| Typical evidence | Probes, observed difficulty, interpretation and response selection | Contact outcomes, timing, routing errors, device and data-transfer problems |
| Participants | People with experiences and language backgrounds relevant to the questions | Cases and field conditions that exercise the planned operation |
| Changes tested | Wording, recall period, examples, instructions and answer categories | Recruitment, consent, staffing, instruments, supervision and processing |
| Main limitation | Does not estimate population outcomes or operational capacity by itself | Smooth delivery can still conceal misunderstanding |
Questions
Start With the Questions Most Likely to Fail
List uncertain terms, sensitive items, long recall tasks and questions feeding important indicators. Ask a person from the intended respondent group to answer, then explore what they understood, how they recalled the information and how they selected an answer. Use neutral probes such as “What does this phrase mean here?” rather than explaining the intended answer first.
Record the question version, the problem, the affected group and the proposed correction. Do not turn a handful of interviews into a percentage supposedly describing the whole population. Cognitive interviewing is a diagnostic exercise. Seek variation that could reveal different interpretations, then check revisions with further participants rather than assuming the first repair worked.
Test the Answer Options as Well as the Wording
For a single-choice question, check whether the categories cover the intended answers without overlapping. Keep different concepts separate: “not applicable,” “do not know,” refusal and a genuine zero do not mean the same thing. Decide how each will be recorded and analyzed.
For a multiple-choice item, make the instruction explicit and test contradictory combinations. For a scale, check that people understand the anchors, the direction and the reference period. Ask whether an apparent midpoint expresses neutrality, mixed experience or lack of knowledge. Where a question concerns household income or another difficult quantity, examine whether the respondent has the information and whether ranges change what can be analyzed.
Test Every Language in Which People Will Respond
A fluent translation can preserve grammar while changing the concept, time period or strength of an answer category. Arrange bilingual review of the intended meaning, resolve disagreements and pretest the resulting wording with target-language respondents. Document changes to the source and each language version together.
AI can suggest a draft translation or alternative phrases. Treat that output as unverified text. It does not establish conceptual equivalence, cultural suitability or measurement validity. Do not send identifiable respondent material to an external AI service. Where an established instrument has permission or adaptation requirements, follow those requirements before changing it.
Pilot and Decision
Run a Pilot That Exercises the Actual Process
Use the proposed consent script, interviewer instructions, device setup and data-handling procedure. Include relevant operating conditions: weak connectivity, different interview languages and respondents who should follow different branches. Synthetic test records can exercise rare routes; keep these distinguishable from real pilot interviews.
Walk a record from first contact through analysis-ready export. Check whether a declined interview is retained as a contact outcome, whether an inapplicable module is skipped correctly and whether a refusal stays distinct from a blank. Examine interview duration alongside interruption, comprehension and respondent burden, rather than setting an arbitrary speed target.
Keep a defect log with severity, owner, correction and retest evidence. A consent or confidentiality failure can block release even when the rest of the questionnaire works. A broken branch requires retesting all routes affected by its repair, not simply reopening the first screen.
A Hypothetical Readiness Decision
Imagine a household survey asking whether a caregiver used a service “recently.” During pretesting, participants interpret this as either the previous week or the previous year. The team replaces it with the defined reference period needed for analysis and checks the new wording in each interview language.
In the pilot, a caregiver reporting no service use still receives a detailed satisfaction module. The team corrects the routing and tests both the user and nonuser paths. A successful upload alone would have missed both problems. Neither exercise establishes the program's service-use rate; the purpose is to decide whether the collection process is ready.
Decide Whether to Revise, Retest or Start
A practical release note identifies the final instrument version, tested languages and modes, unresolved limitations, and the person responsible for the decision. Link each serious issue to evidence that the correction worked. Keep the pilot data separate from the main survey unless the design explicitly permits combining them and comparability has been assessed.
The scope should fit the risk and available resources. No single exercise proves that every question is valid or every field problem is solved. Where testing coverage is incomplete, state what remains uncertain and what additional checks will happen during supervised early fieldwork.
FAQs
Frequently Asked Questions
Does a Pilot Replace Cognitive Pretesting?
Not automatically. A field pilot may show that interviews finish and files upload while missing a shared misunderstanding. Include methods that examine how respondents understand and answer the questions.
How Many Participants Are Needed?
There is no universal number for every pretest or pilot. Choose participants and coverage for the problems, languages, routes, modes and field conditions being tested, and document what the exercise cannot establish.