Difference-in-Differences

Difference-in-differences (DiD) estimates the effect of an intervention by comparing how a treated group changed between two periods with how a comparison group changed over the same periods. The estimate is the treated group’s change minus the comparison group’s change.

The calculation

Take average pull requests per developer-week for a group of teams that adopted a tool (the treated group) and for teams that did not (the comparison group), before and after adoption.

Adopters were ahead before and are ahead after, but that gap of 0.24 existed before the tool. Adopters also improved, but so did everyone else. A before-and-after comparison of adopters alone would report +0.16, and an after-only comparison would report +0.26. Subtracting the comparison group’s change removes both the permanent gap and the change that both groups shared, leaving +0.02.

The assumption

DiD identifies the effect only under parallel trends: without the intervention, the treated group would have changed by the same amount as the comparison group. The comparison group stands in for the treated group’s unobserved counterfactual change, so the assumption is about something that cannot be observed. The simulation below shows what happens when it holds and when it fails. Each of 14 treated and 16 comparison teams has its own baseline, and everyone drifts up by 0.10. Python 3.12.14, numpy 2.2.3; values are means over 2,000 simulated studies.

import numpy as np
# hand example
t_pre, t_post, c_pre, c_post = 1.89, 2.05, 1.65, 1.79
print("hand: DiD =", round((t_post-t_pre)-(c_post-c_pre),2))
rng = np.random.default_rng(11)
def sim(extra_trend, effect=0.0, reps=2000, n=14, m=16):
    out=[]
    for _ in range(reps):
        base_t = 1.9 + rng.normal(0,.15,n); base_c = 1.65 + rng.normal(0,.15,m)
        drift = 0.10                      # shared drift
        pre_t = base_t + rng.normal(0,.05,n); post_t = base_t + drift + extra_trend + effect + rng.normal(0,.05,n)
        pre_c = base_c + rng.normal(0,.05,m); post_c = base_c + drift + rng.normal(0,.05,m)
        out.append(((post_t-pre_t).mean() - (post_c-pre_c).mean()))
    return np.mean(out)
print("true effect 0, parallel trends hold      : DiD mean = %.3f" % sim(0.0))
print("true effect 0.10, parallel trends hold   : DiD mean = %.3f" % sim(0.0, 0.10))
print("true effect 0, treated drifts +0.08 extra: DiD mean = %.3f" % sim(0.08))
# scale: multiplicative growth, same % change but different levels
t0,c0=2.0,1.5; g=0.10
print("scale: both groups grow 10%% with no effect: levels DiD = %.3f, log DiD = %.3f" % (((t0*(1+g)-t0)-(c0*(1+g)-c0)), np.log(t0*(1+g)/t0)-np.log(c0*(1+g)/c0)))

hand: DiD = 0.02
true effect 0, parallel trends hold      : DiD mean = 0.000
true effect 0.10, parallel trends hold   : DiD mean = 0.100
true effect 0, treated drifts +0.08 extra: DiD mean = 0.079
scale: both groups grow 10% with no effect: levels DiD = 0.050, log DiD = 0.000

When the trends are parallel, the estimate is 0.000 under no effect and recovers 0.100 under a true effect of 0.10, even though the groups started at different levels. When the treated teams were already drifting up an extra 0.08 for unrelated reasons, the method reports 0.079 for a tool that does nothing. Nothing in the estimate warns of this.

Checking and limits

• Look at the periods before treatment. If the groups moved together over several earlier periods, parallel trends is more believable. Similar pre-trends are supporting evidence, not a proof, since the question concerns the post-period counterfactual.
• Nothing else may change for one group only. A reorganization, a holiday, or a campaign that hits one group at the same time is attributed to the intervention.
• The scale matters. Parallel trends in levels and in percentages are different assumptions. In the last line of the output, two groups at 2.0 and 1.5 both grow 10 percent with no effect: the DiD in levels is 0.050 while the DiD in logs is 0.000. Choose the scale on which the groups would plausibly have moved together, and say which.
• Selection into treatment on trend breaks it. If the groups that adopt are those already improving faster, as volunteers often are, the method overstates the effect.
• Staggered adoption needs care. When units adopt at different times, simple two-way fixed-effects regressions can use already-treated units as comparisons and give misleading estimates.

Uncertainty is also larger than the arithmetic suggests. The unit of analysis should be the unit that was assigned or that shares shocks, such as the team, and the interval should reflect the variation between those units. Randomly assigning the intervention, when possible, removes the need for the parallel-trends assumption. See causal effect for the conditions this method replaces and causal impact for the time-series counterpart.

References

• Scott Cunningham, Causal Inference: The Mixtape, Difference-in-Differences