At a Glance
A controlled before-after study measures a program group and a comparison group both before and after a program. Difference-in-differences (DiD) is the calculation that turns those four measurements into an effect estimate. If you have the data for one, you have the data for the other, so the question is how much you claim from it.
At a Glance
| Controlled before-after | Difference-in-differences | |
|---|---|---|
| What it is | A study design | An estimate (and its assumption) |
| Data needed | Outcome before and after, program and comparison group | The same four numbers |
| Core move | Compare the change in each group | Subtract the comparison group's change from the program group's change |
| What it must state | Why the comparison group is a fair stand-in | That both groups would have changed alike without the program (parallel trends) |
| Typical use of the label | Health services and public health research | Economics and impact evaluation |
| Main weakness | Groups may be on different paths for reasons other than the program | The same, plus too few time points to test it |
The two labels grew up in different fields, so reports often use them for the same study. Treat the table as a guide to emphasis, not a strict boundary. Both belong to the family of quasi-experimental designs, and both try to build a counterfactual without randomization.
Two-by-Two Example
A Two-by-Two Worked Example: Reading Corners
A fictional program, Reading Corners, sets up book areas in 20 primary schools. Twenty similar schools in the same district get nothing. Pupils take the same reading test (scored 0 to 100) before the program and again after two years. The numbers are made up for the example.
| Average reading score | Before | After | Change |
|---|---|---|---|
| Program schools | 40 | 52 | +12 |
| Comparison schools | 44 | 49 | +5 |
| Difference between groups | -4 | +3 | +7 |
The +7 in the bottom right is the difference-in-differences: the program schools changed by 12 points, the comparison schools by 5, and 12 minus 5 is 7. You get the same answer from the other direction: the gap between groups moved from -4 before to +3 after, a swing of 7 points.
Three ways to read the same table, and what each one gets wrong:
| Reading | Calculation | Result | Problem |
|---|---|---|---|
| After only | 52 minus 49 | +3 | Ignores that program schools started 4 points behind |
| Before-after, program group only | 52 minus 40 | +12 | Counts the 5 points every school gained anyway, from a new curriculum, ordinary growth or a test that gets easier with practice |
| Difference-in-differences | (52 minus 40) minus (49 minus 44) | +7 | Valid only if program schools would have gained 5 points without Reading Corners |
An estimate of +7 should come with a margin of error from the data (the 40 schools, with pupils clustered inside them), not only the point estimate. See How to Choose Sample Size for the planning side.
Checking Parallel Trends
The assumption that holds the +7 together cannot be proved, but it can be tested against the past. Suppose the district also tested pupils two years before the program began.
| Average reading score | Two years before | Before | Change |
|---|---|---|---|
| Program schools | 36 | 40 | +4 |
| Comparison schools | 40 | 44 | +4 |
Both groups rose by 4 points in the earlier period, so they were moving together before the program, and the assumption is more believable. Had the program schools been rising by 8 points while the comparison schools rose by 4, the +7 would partly reflect a head start that Reading Corners did not create. A single before and after measurement cannot show this, which is the main reason to collect a second pre-program round when you can. Matching on school size or past scores helps, but it does not replace the check.
Which Label to Use
Which Label Should You Use?
Use controlled before-after when you are describing the design to a general or health audience, or when you only have two rounds and want to be modest about what it proves. Use difference-in-differences when you are reporting an effect estimate, naming the parallel trends assumption and showing the pre-program evidence. In a report, say both once: "a controlled before-after design, analyzed with difference-in-differences."
If you can assign schools at random, the design changes: see RCT vs quasi-experimental design. If you have no comparison group at all, you have a plain before-after comparison, which cannot separate the program from everything else that changed.
Common Mistakes
Calling a before-after comparison "controlled" when the comparison group is not comparable. If the schools were chosen because they needed the program most, the groups may be on different paths. Describe how the comparison group was chosen and what differs.
Reporting only the point estimate. Give the uncertainty and the pre-program evidence, or the reader cannot judge the +7.
Ignoring what else changed in only one group. A new teacher training scheme in just the program schools would be mixed into the +7. Ask the field team what else arrived at the same time.
Adjusting for things the program changes. Controlling for post-program variables can remove part of the effect. See confounders, mediators and colliders.
Choosing the label by what sounds more rigorous. The rigor is in the comparison group, the number of time points and the checks you report, not in the name.