At a Glance
A survey estimate has two parts: a number and a statement of how precise it is. Both depend on how the sample was drawn. This page works one fictional survey end to end, so you can see where sampling weights, the design effect and the confidence interval enter and what each changes. All numbers are made up.
At a Glance
| Step | Question | What It Fixes | If Skipped |
|---|---|---|---|
| Weights | Did every household have the same chance of selection? | Over-representation of heavily sampled groups | The estimate describes the sample, not the population |
| Stratified standard error | How much does the estimate vary between samples drawn the same way? | The calculation that matches the design | Precision is judged on the wrong formula |
| Design effect | Are interviews in a cluster alike? | Intervals that are too narrow | The result looks more exact than the data allow |
| Comparison with the target | Is the whole interval above, below or around the target? | Over-confident conclusions | A point estimate drives the decision |
Worked Case
Worked Case: A Coverage Survey in Two Strata
A district wants to know what share of households use a safe water source. The target set before the survey is 60%. The district has two strata:
| Urban | Rural | |
|---|---|---|
| Households in the stratum | 8,000 | 24,000 |
| Share of all households | 25% | 75% |
| Households interviewed | 300 | 300 |
| Using a safe source | 225 (75%) | 150 (50%) |
| Base weight | 8,000 / 300 = 26.7 | 24,000 / 300 = 80 |
The team sampled equal numbers so that both strata can be reported on their own. Within each stratum, households were chosen in clusters of 15, so each stratum has 20 clusters. This assumes a complete household list, full response among those selected, and accurate answers; none of these is checked here.
Step 1: The Weighted Estimate
The share of respondents using a safe source is 375 / 600 = 62.5%. That figure treats every household as equal, but one rural household stands for three times as many district households as one urban household.
The weighted estimate is the sum of each stratum share times its population share: 0.25 x 75% + 0.75 x 50% = 56.25%.
The weights give the same figure: (26.67 x 225 + 80 x 150) / (26.67 x 300 + 80 x 300) = 18,000 / 32,000 = 56.25%.
The unweighted result is 6.25 points higher because the sample over-represents the better-served urban stratum. The unweighted number answers "what share of the people we interviewed use a safe source?", not the district question.
Step 2: The Standard Error
For a stratified sample, the variance of the estimate adds each stratum's variance, scaled by its population share squared:
Variance = 0.25² x (0.75 x 0.25 / 300) + 0.75² x (0.50 x 0.50 / 300) = 0.0000391 + 0.0004688 = 0.000508.
The standard error is the square root: 2.25 points. A 95% interval from this alone is 56.25 plus or minus 1.96 x 2.25, which is 51.8% to 60.7%. The formula treats the 600 households as independent draws within each stratum, which is the next thing to check.
Step 3: Allow for Clustering
Households in the same village or cluster resemble each other. Assume an intraclass correlation of 0.05 for this outcome (an illustrative value; see intraclass correlation for how to source one) and 15 households per cluster. The design effect from clustering is 1 + (15 - 1) x 0.05 = 1.7.
The standard error grows by the square root of 1.7, which is 1.30. The new standard error is 2.25 x 1.30 = 2.94 points, and the 95% interval is 56.25 plus or minus 5.76, which is 50.5% to 62.0%.
The design effect applied here is an approximation. It is assumed to be the same in both strata, to rest on equal-sized clusters and to be layered on the stratified formula. With only 40 clusters, a t value of about 2.02 instead of 1.96 would widen the interval slightly, to about 50.3% to 62.2%. Real surveys estimate the variance directly from the cluster data, using survey software that knows the strata, clusters and weights.
| Calculation | Estimate | 95% Interval | Includes the 60% Target? |
|---|---|---|---|
| Naive: no weights, treat as one random sample | 62.5% | 58.6% to 66.4% | Yes |
| Weighted, stratified standard error only | 56.3% | 51.8% to 60.7% | Yes |
| Weighted, stratified, with design effect 1.7 | 56.3% | 50.5% to 62.0% | Yes |
What the Result Supports
The naive calculation reports 62.5%, with an interval that sits mostly above the 60% target. The design-based calculation puts the best estimate at 56.3%, with an interval running from 50.5% to 62.0%. The naive interval does not even contain the weighted estimate, so it points to a different part of the range altogether.
The honest summary is: "We estimate that 56% of households use a safe source (95% CI 50.5% to 62.0%). The data are compatible with being near the 60% target and with falling several points short. They cannot settle whether the target was met." If the decision needs a clear answer, plan a larger sample or more clusters; see how to choose sample size. The interval also covers sampling variation only. If the household list missed informal settlements or many selected households did not answer, the estimate may be off in ways the interval does not show. See nonresponse bias and sampling frame.
To judge whether a gap from a target matters in practice, set the threshold first, as in statistical vs practical significance.
Reporting It
What to Put in the Report
The estimate and its interval, with the confidence level: "56.3% (95% CI 50.5% to 62.0%)".
The design, in one line: "stratified two-stage cluster sample, 300 households per stratum, 15 per cluster, weighted to stratum size".
The sample sizes, unweighted: 600 households, 40 clusters.
The assumptions, including the source of the intraclass correlation or design effect.
The limits, covering coverage of the list, nonresponse and measurement.
The comparison with the target, in words that match the interval.
Check an AI-drafted results section against this list. Ask which weight variable was used, whether strata and clusters were declared in the variance calculation, and recompute one figure yourself.
Common Mistakes
Reporting the share of respondents as the population share. This is correct only when selection chances were equal and response was even.
Computing the interval with the simple formula after weighting. Weights alone do not make the standard error right. The strata and clusters must be in the variance calculation.
Forgetting that subgroups have their own precision. An interval for the whole district is not an interval for a ward. Subgroups with few clusters or interviews have much wider ones; see disaggregation.
Treating a point estimate as the finding. A figure of 56.3% against a target of 60% is not a miss when the interval spans the target.
Calling a narrow interval accurate. Precision is not accuracy. A tight interval around a biased estimate is wrong with confidence.