At a Glance
"How many do we need to survey?" has at least three different answers, depending on what the survey must support. This page helps you pick the right question first. It uses a single fictional indicator. You should know what a confidence interval is; the formulas for simple surveys are in how to choose sample size.
At a Glance
| Precision | Power | Classification | |
|---|---|---|---|
| Decision it supports | Report a level with stated margin | Detect whether a change of a given size happened | Label an area above or below a target |
| Inputs | Margin of error, confidence level, variability | Size of change, significance level, power, variability | Lower and upper performance levels, acceptable error at each |
| Reads as | "Coverage is 60%, plus or minus 5 points" | "If coverage truly rose 10 points, we would detect it 80% of the time" | "A district passing this rule is above target, with a stated error risk" |
| Fails when | Used to claim a change was detected | Used after the data are in, or when the comparison is not credible | Used to estimate the district's coverage |
| Sample grows when | The margin narrows | The change to detect shrinks | The two levels move closer together |
Worked Case
Worked Case: One Indicator, Three Questions
A fictional program reports the share of households using a safe water source. At baseline the figure is about 40%. The program expects a rise to 50%. It also wants a coverage level for the region, and wants to find which of its supervision districts fall below an agreed target.
Assume simple random samples and full response. Clustering is added where it matters. All calculations keep full decimals and round up once at the end.
Question 1: How Precisely Is Coverage Known?
The goal is a margin of error of plus or minus 5 points at 95% confidence. With the most cautious assumption of 50% for the proportion, the sample is z² x 0.25 / 0.05², where z is 1.95996, which is 384.15. Round up once to 385 households per estimate. If the design effect from clustering is 1.5, the unrounded 384.15 x 1.5 = 576.22 becomes 577.
This sample says how well a level is known. It says nothing yet about changes.
Question 2: Can We Detect a Change?
The goal is to detect a rise from 40% to 50% with 80% power at a 5% two-sided significance level, comparing two independent samples. The standard formula is:
n per group = (z for 95% x square root of (2 x p-bar x (1 - p-bar)) + z for 80% power x square root of (p1 x (1 - p1) + p2 x (1 - p2)))² / (p1 - p2)²
With p1 = 0.40, p2 = 0.50, p-bar = 0.45, z values of 1.95996 and 0.84162, this gives 387.34, rounded up to 388 per group. With a design effect of 1.5 it is 581.01, so 582 per group.
The minimum detectable effect idea (Bloom 1995) turns the question around: for a given sample, what is the smallest true change the study would detect with the chosen power? For 385 per group and a baseline of 40%, it is about 10.0 points.
Same Sample, Different Statements
388 and 385 are close, which invites the mistake of treating them as the same thing. Take 385 households in each of two rounds.
| What | Result |
|---|---|
| Half-width of each level, at 50% | plus or minus 5.0 points |
| Half-width of the difference between 40% and 50% | plus or minus 7.0 points |
| Power if the true change is 5 points | 28.9% |
| Power if the true change is 7.5 points | 55.5% |
| Power if the true change is 10 points | 79.8% |
| Power if the true change is 12.5 points | 93.7% |
| Power at 10 points, with design effect 1.5 | 62.5% |
A survey precise enough to give each level to within 5 points has an interval of about plus or minus 7 points for the change, because the difference of two estimates is more variable than either. If the program's real effect is 5 points, this study will usually miss it. Precision for a level is not power for a change, and neither is a guarantee that the change was caused by the program. Whether the later group differs from the earlier one for reasons other than the program depends on the design, not on the sample size. A larger sample does not repair a comparison between groups that were selected differently.
Question 3: Which Districts Fall Below the Target?
Now the program needs to flag districts for supervision, with 19 households sampled in each. An area passes if at least 15 of the 19 use a safe source. This is a lot quality assurance sampling (LQAS) rule, a classification method described in the LQAS entry and reviewed by Robertson and Valadez (2006). The chance that a district passes depends on its true coverage:
| True Coverage | Chance of Passing |
|---|---|
| 50% | 1.0% |
| 60% | 7.0% |
| 70% | 28.2% |
| 80% | 67.3% |
| 90% | 96.5% |
If the lower level is 50% and the upper level 90%, the rule rarely passes a poor district (1.0%) and rarely fails a good one (3.5%). In between, the label is not reliable: a district at 80% passes only about two times in three. The sample of 19 gives an interval of about plus or minus 22 points for any one district's coverage. It is a good classifier for districts far from the threshold, and a poor estimator for any single one. Do not publish district coverage figures from this sample, and do not use the rule if households are sampled in clusters or with unequal probability without recomputing the error rates.
Limits and Checks
After the Data Are In
Once results are in, power calculations built from the observed difference add no information. The confidence interval already shows which sizes of change the data are compatible with (Goodman and Berlin 1994; Hoenig and Heisey 2001). A nonsignificant result with a wide interval means the study could not settle the question; it does not show there was no change. For wider cautions on interpreting intervals and power, see Greenland and others (2016) and statistical vs practical significance.
What the Calculation Assumes
- Simple random sampling. Clustered or weighted samples need the design effect, see design effect.
- A comparison that estimates the effect. Power describes the chance of detecting a difference, not whether that difference reflects the program. Assignment and selection are separate questions.
- A rough baseline value. The 40% is assumed. A baseline different from 40% changes the variability and the sizes.
- Full response. Divide by the expected response rate afterward, and see nonresponse bias.
- One comparison. Subgroups or several indicators each need their own calculation; see disaggregation.
Check an AI Sample Size
A fictional AI draft says: "With 385 households per group, the study has 95% confidence and 80% power to detect a 5-point change." The checks:
Identify which question it answers. 385 per group is the answer to precision at plus or minus 5 points for a level, not to power.
Recompute power from the stated inputs, with a spreadsheet, R or Python. For 40% to 45% at 385 per group it is 28.9%, not 80%. The tool should say which formula, which tails and which significance level it used.
Simulate it. Draw many pairs of samples from 40% and 45%, apply the test and count how often it rejects. For this case, 20,000 simulated studies gave 29.7% power at 5 points and 79.8% at 10 points, consistent with the formulas.
Test the neighbors. Change the inputs and confirm the answer moves in the right direction: smaller change, larger sample.
Record the model, version and date, and any change you made.
People still decide the smallest change worth detecting, which comes from program knowledge and not from the formula.
Common Mistakes
Choosing a sample for a margin of error and then claiming it detects change. The two questions give nearby numbers here by coincidence.
Reporting a power figure after the study. Report the interval.
Treating a nonsignificant result as no difference. The interval may include important changes.
Sizing for one comparison and analyzing many. Each subgroup or indicator needs its own sample.
Using a classification sample to publish estimates. Small LQAS samples can label areas; they cannot estimate coverage within a useful margin.
Forgetting the design effect. A design effect of 1.5 takes power at a 10-point change from 79.8% to 62.5% in this case.