At a Glance
Statistical significance says a difference is unlikely to be chance alone. Practical significance asks whether the difference is big enough to matter for the people, the budget or the decision. The first depends heavily on sample size, the second on judgment, so a report needs both. When you judge how much weight a published claim deserves, a meta-evaluation looks at the quality of the evaluation itself.
At a Glance
| Statistical significance | Practical significance | |
|---|---|---|
| Question | Could this difference be chance? | Is this difference big enough to matter? |
| Evidence | p-value, confidence interval | Effect size, cost, benefit, what people would notice |
| Depends on sample size? | Strongly. Large samples find tiny differences | No. A difference is the same size in any sample |
| Who decides | The analysis, by convention (often 0.05) | The program team, funder and affected people, agreed in advance |
| Can mislead when | Sample is very large or very small | The threshold is set after seeing the results |
| Report it as | p-value with the interval | Effect in real units next to the agreed meaningful change |
The statistical significance entry explains the test itself. This page shows how to place the result next to a judgment about size.
Worked Case
Worked Case: Two Tutoring Studies
A fictional ministry asks two teams to test small-group tutoring. Before the studies, the ministry says a gain of at least 3 points on the 100-point reading test, about 0.2 standard deviations, is the smallest gain that would justify the cost of rolling the program out. Test scores in both studies have a standard deviation of 15 points. All numbers are made up for the example.
| Study A: district-wide | Study B: pilot | |
|---|---|---|
| Pupils | 4,000 tutored, 4,000 not | 30 tutored, 30 not |
| Average gain over comparison | 0.9 points | 6 points |
| Effect size (gain divided by 15) | 0.06 | 0.40 |
| p-value | about 0.007 | about 0.13 |
| 95% confidence interval | 0.2 to 1.6 points | -1.8 to 13.8 points |
| Statistically significant at 0.05? | Yes | No |
| Reaches the 3-point threshold? | No, the whole interval is below 3 | Maybe. The interval spans 0 to well above 3 |
How to Read the Two Results
Study A is significant and not important. With 8,000 pupils, even a gain under one point is unlikely to be chance. The best estimate is 0.9 points and the data rule out a gain of 3 or more. A headline saying "tutoring significantly improves reading" would be true and would mislead, because the gain falls short of what the ministry said it needed.
Study B is not significant and not a verdict of "no effect". Sixty pupils cannot estimate a gain precisely. The best estimate, 6 points, would be a large gain, but the interval runs from a small loss to an enormous gain. The honest summary is "inconclusive: promising, and too small to tell". Reporting "no significant effect" would send readers the wrong way.
The practical decision differs too. For A, stop or redesign: the program works a little and the cost is not justified at that size. For B, run a larger study (see deciding how many to interview) to see whether the gain is real, aiming for a precision that can separate 0.9 from 3.
The pattern generalizes. Large samples tend to produce results that are significant and small. Small samples tend to produce results that are large and not significant. The interval is the part that tells you which situation you are in.
Reporting It
What to Put in the Report
Give readers three things for each important result, not only the p-value:
- The estimated effect in real units. "A 0.9-point gain on the reading test", not "a significant improvement".
- The interval. It shows how precisely the study measured the gain.
- The meaningful threshold and where it was set. "The ministry said at least 3 points before the study, from the cost of rolling out."
Then write the conclusion in plain words: "The gain is real and small" (A), or "The study cannot tell" (B). The findings, conclusions and recommendations page covers how to keep the claim tied to the evidence. For where results sit in the report, see evaluation report, and for the numbers behind a design, see quantitative data.
Common Mistakes
Setting the meaningful difference after seeing the results. It is tempting to pick a threshold the result just clears. Write it down before the analysis.
Treating 0.05 as a pass mark for importance. The cutoff is a convention for how surprising a result is under a "no difference" assumption. It says nothing about value.
Reading a non-significant result as proof of no effect. An absence of evidence is not evidence of absence, especially with a small sample.
Reporting the effect size without a sense of scale. An effect size of 0.2 means different things for a reading test and a mortality rate. Translate to units people use, such as points, days or dollars.
Ignoring cost. A gain that is small and cheap can be worth scaling. The same gain that is expensive may not be. Practical significance includes what it takes to get the effect.
Testing many things and reporting the hits. If you check 20 outcomes at the 0.05 level, one will often appear significant by chance. State which outcomes you planned to test.