At a Glance
Reliability is whether a measure gives the same result when nothing has changed. Validity is whether it measures what you say it measures. A good indicator needs both, and they fail in different ways, so they need different checks.
At a Glance
| Reliability | Validity | |
|---|---|---|
| Plain question | Would I get the same answer again? | Is this the right thing to measure? |
| What goes wrong | Random error: results swing between people, days or forms | Systematic error: results are consistent but aimed at the wrong target |
| Typical causes | Vague questions, untrained enumerators, no scoring guide | Concept not covered, proxy that does not follow the concept, bias in who answers |
| Common checks | Test-retest, agreement between scorers, internal consistency | Expert review of content, comparison with an accepted measure, tests of expected patterns |
| Usual statistic | Percent agreement, Cohen's kappa, intraclass correlation, Cronbach's alpha | Correlation with a reference measure, group differences you would predict |
| Fix | Clearer wording, a rubric, training, more items | Redefine the concept, change the measure, add a source |
| Can you have it without the other? | Reliable and not valid: yes | Valid but not reliable: only on average, not for single results |
The entries for reliability and validity define each one. This page puts them side by side and shows what each looks like when it fails.
Reliability Example
Example 1: A Reliability Problem
A fictional education program wants to know how often teachers use group work in class. Two observers visit 20 classrooms, watch the same lesson in each, and rate group work as "none", "some" or "a lot".
| Round | Classes both observers rated the same | Percent agreement |
|---|---|---|
| First visit, no scoring guide | 11 of 20 | 55% |
| After a one-page rubric and a half-day training | 18 of 20 | 90% |
At 55 percent agreement the rating depends heavily on who happened to visit. The measure might still be a good idea (group work does matter), but you cannot tell whether a change from one year to the next is real or a different observer. After the rubric the same measure gives nearly the same answer for each class, so it is reliable enough to compare years. Percent agreement is the simplest check. Kappa corrects it for agreement by chance.
Fixing reliability did not prove the rating is right. Both observers may still be rating how noisy the class is, not how much group work happens. That is a validity question, which the second example takes up.
Validity Example
Example 2: A Validity Problem
A fictional livelihoods program builds a short women's empowerment index. One of the questions, chosen because it is quick to ask, is "Does a woman in this household own a mobile phone?" Asked twice a week apart, nearly every household gives the same answer. By any consistency test the item is reliable.
The program then checks it against longer interviews with 30 of the women, which ask who decides about spending, health care and visiting family. Women who own a phone were not noticeably more likely to report taking those decisions. In several households a husband bought the phone and kept control of it.
| Check | What the program found |
|---|---|
| Test-retest (same women, a week apart) | Same answer in 98% of households. Reliable |
| Content (does it cover decision-making?) | No. It covers one asset |
| Comparison with the longer interviews | Little link with decision-making. Weak validity as an empowerment measure |
The phone question may still be a fine indicator of phone ownership. It is not a valid stand-in for empowerment. Because the item was consistent, its scores looked trustworthy in a table. Reliability can make a bad proxy look good.
What to Check
How the Two Relate
Low reliability puts a ceiling on validity: if a measure swings at random, it cannot track anything well. High reliability puts no floor under validity. So check in this order:
Define the concept in words first. What exactly should this indicator show? Write it in the indicator reference sheet.
Test reliability. Use a pilot (pretest vs pilot) or a small repeat measurement. Fix vague wording and train scorers.
Test validity. Compare with a better measure where one exists, even on a small sample. Ask people who know the context whether the items cover the concept.
Record what you did and what you could not check. Some indicators will be accepted on judgment alone. Say so.
The wider quality picture, including accuracy, timeliness and completeness, is in five data quality dimensions. For questionnaires, see how to design a questionnaire.
Common Mistakes
Reporting reliability as if it proved quality. An alpha of 0.9 says the items agree with each other. It does not say they measure the right concept. Very high alpha can even mean the items are repetitive.
Using one word for both. "The data is not accurate" can mean two different problems. Decide whether the numbers vary at random or are consistently off, because the fix is different.
Skipping validity because a measure is widely used. A measure built in another country, language or age group may not fit yours. Test it locally, or at least with a few people who know the context.
Judging validity by looks alone. A question can seem sensible to the team and still be understood differently by respondents. Cognitive pretesting finds this faster than analysis does.
Treating small samples of checks as proof. Twenty classes or thirty interviews show a problem clearly. They do not guarantee quality everywhere, and the sample for the check should match the situations you plan to use the measure in.