Unequal Cluster Sizes: Effective Sample Size When Clusters Differ

Worked Case: Ten Schools of Different Sizes

A fictional program surveys all pupils in grade 4 in ten schools, chosen at random from a list of schools. Selection is by school; no school was assigned to anything. The outcome is whether a pupil reads at grade level.

There are 302 pupils, so the average school has 30.2. The sum of squared sizes is 14,094. Assume rho = 0.10, an illustrative value; the intraclass correlation page and Evaluation Review (2023) describe how to find values for your own outcome and setting. Assume also that the same rho applies to every school.

Step 1: The Equal-Size Shortcut

Using the average size of 30.2, the design effect is 1 + (30.2 - 1) x 0.10 = 3.92. The effective sample is 302 / 3.92 = 77.0 pupils, and the standard error is 1.98 times as large as under simple random sampling.

Step 2: Allow for Unequal Sizes

Large schools carry more of the weight, and pupils in one school resemble each other, so a few big schools make the average less stable. The design effect for an unweighted average across all pupils uses the sum of squared sizes divided by the sum of sizes: 14,094 / 302 = 46.67. The design effect is 1 + (46.67 - 1) x 0.10 = 5.57.

The effective sample is 302 / 5.57 = 54.2 pupils, and the standard error is 2.36 times as large as under simple random sampling. The shortcut said 77; the actual figure is 54. The same quantity is often written 1 + ((CV² + 1) x m - 1) x rho, with CV the coefficient of variation of cluster size (Eldridge and others 2006). With the population form of CV (0.738 here) it gives the same 5.57; with the sample standard deviation (0.778) it gives 5.75. Check which convention a source uses before mixing results.

The gap grows with rho. If your intraclass correlation is uncertain, show the range rather than a single value.

Step 3: Effective Sample Size and Effective Clusters

The two largest schools hold 140 of the 302 pupils, 46.4% of the data. The effective number of clusters is 302² / 14,094 = 6.5. The sample has ten schools but behaves, for this purpose, like six or seven equal ones. This number is a descriptive guide, not a formal degrees-of-freedom count. It shows that the result leans on a few places, and that each should be examined before the estimate is trusted.

Step 4: What It Does to an Interval

Suppose 40% of pupils read at grade level. Under simple random sampling the standard error would be the square root of (0.40 x 0.60 / 302), which is 2.82 points, and the 95% half-width is 5.5 points.

With ten schools, a normal multiplier of 1.96 is too small; a t value of 2.26 for nine degrees of freedom is more honest, and even that is approximate. Survey software that declares the school as the cluster estimates the variance directly, and should replace these hand calculations in a real analysis.

Step 5: Plan the Number of Schools

Suppose the program needs an effective sample of 150 pupils. If a new survey has the same mix of school sizes, the design effect stays at 5.57, so it needs 150 x 5.57 = 835.0 pupils. At 30.2 per school that is 27.65, so 28 schools. The equal-size shortcut gives 150 x 3.92 = 588.0 pupils and 19.47 schools, so 20. The shortcut would have under-planned by eight schools, or 40%.

Rounding happens once, at the end, as in how to choose sample size. Where the survey can choose its clusters, two things reduce the penalty: sampling clusters of similar size, and capping interviews in the largest clusters. Capping makes weights unequal, so recompute with both effects.

When Weights Also Vary

Unequal weights add a second source of variance inflation. A common approximation for weights alone is n x sum of squared weights / (sum of weights)². With 100 respondents at weight 1 and 100 at weight 3, that is 200 x 1,000 / 160,000 = 1.25. When clustering and weights both matter, the two factors are often multiplied as an approximation: here, 1.25 x 5.57 = 6.96. This is a planning shortcut; survey software computes the combined effect from the data. See sampling weights.

What a Design Effect Does Not Fix

A design effect changes the variance of an estimate. It does not change where the estimate sits. If the school list omits private schools, if absent pupils differ from present ones, or if the reading test is poorly translated, the estimate is off by an amount no design effect corrects. A wider interval around a biased estimate is still around the wrong value. See nonresponse bias.

The same arithmetic applies when clusters are assigned to a program rather than sampled. Random assignment of schools to a program is a different step from drawing schools for a survey, and each has its own sample size question. In a cluster-randomized trial, unequal cluster sizes can lower power; Eldridge and others report that the effect is negligible when the coefficient of variation is below 0.23.

Other limits of the calculation:

• It assumes one rho for all clusters and equal variances. A rho that varies by school type changes the result.
• It is for a simple unweighted average. Cluster-level analyses, ratios and subgroup estimates behave differently.
• The cluster sizes are treated as fixed. If they are themselves random, the effect differs.
• Subgroups such as girls or rural schools have their own, smaller effective sample.

Check an AI Calculation

An AI draft might say: "With an average of 30 pupils per school and rho = 0.10, DEFF is 3.9, so 20 schools give an effective sample of 150." The checks:

1. 1

   Ask for the cluster sizes it used. If it used only the average, the design effect has ignored the spread.

2. 2

   Recompute the sum of sizes, the sum of squared sizes and their ratio from the table. Expect 302, 14,094 and 46.67.

3. 3

   Simulate it. Generate data with a shared school effect (here rho = 0.10), compute the unweighted mean across many repeats and compare its variance with simple random sampling. For this case, 30,000 simulated surveys gave a design effect of about 5.59, close to the formula's 5.57 and far from 3.92.

4. 4

   Check the source of rho. The draft should name where the value came from, and the passage should support it for a similar outcome and setting.

5. 5

   Record the model, version and date, and what you changed.

The method and the assumed rho remain a human decision. A tool can compute them; it cannot know whether your outcome clusters like the one in a cited study.

Common Mistakes