Intention-to-Treat Analysis

Intention-to-treat (ITT) analysis compares the groups that were formed by random assignment, counting every unit in the group it was assigned to, whether or not it received or used the treatment. In a pilot that offers a tool to half of the teams, every developer on an offered team is analyzed as treated, including those who never opened the tool, and every developer on a control team is analyzed as untreated.

Why not compare the users

Randomization makes the offered and control groups comparable in expectation on measured and unmeasured characteristics, though a finite sample can still have chance imbalances. Take-up is not randomized. People decide for themselves whether to use what they were offered, and the ones who do tend to differ from the ones who do not, often in ways that also affect the outcome. Restricting the offered group to its users and comparing them with the control group replaces a randomized comparison with a self-selected one, which is the problem randomization was meant to remove. Dropping non-users after the fact has the same effect.

The simulation below makes this concrete. Half of 20,000 developers are offered a tool at random. More productive developers are more likely to use it. Using the tool adds 2.0 to the outcome (an arbitrary unit), and the outcome also rises with an unobserved productivity trait. Python 3.12.14, numpy 2.2.3.

import numpy as np
rng = np.random.default_rng(7)
n = 20000
skill = rng.normal(size=n)                     # unobserved: more productive developers
offered = rng.permutation(n) < n // 2          # randomized offer
uses = offered & (rng.random(n) < 1 / (1 + np.exp(-(skill - 0.2) * 2)))   # more productive people take it up
effect = 2.0                                   # true effect of USING the tool
y = 10 + 3 * skill + effect * uses + rng.normal(size=n)

itt = y[offered].mean() - y[~offered].mean()
takeup = uses[offered].mean()
naive = y[uses].mean() - y[~offered].mean()
wald = itt / takeup
print(f"take-up among offered          : {takeup:.3f}")
print(f"ITT (offered - control)        : {itt:.3f}")
print(f"users vs control (as-treated)  : {naive:.3f}")
print(f"ITT / take-up (Wald)           : {wald:.3f}   true effect of using = {effect}")
print(f"true effect of the offer       : {effect*takeup:.3f}")

take-up among offered          : 0.432
ITT (offered - control)        : 0.838
users vs control (as-treated)  : 4.015
ITT / take-up (Wald)           : 1.940   true effect of using = 2.0
true effect of the offer       : 0.864

Take-up is 43 percent. The ITT estimate, the difference in mean outcome between everyone offered and everyone in control, is 0.84. It is close to the true effect of the offer, which is the effect of using the tool times the share who use it: 2.0 × 0.432 = 0.86. Comparing only the users with the control group gives 4.0, twice the true effect of using the tool, because the users were already more productive. ITT is smaller than the effect on a user because most of the offered group is diluted by non-users, and it answers a different question: what happens to an organization that makes the tool available.

What ITT does and does not estimate

ITT estimates the effect of the assignment, here the offer. That is usually what a rollout decision needs, since deciding to roll out means deciding to offer. Its value depends on take-up in the pilot. If reminders raise take-up from 43 to 70 percent, the ITT estimate grows even though the tool is unchanged. A pilot’s ITT therefore transfers to a wider rollout only if the rollout is expected to achieve similar take-up.

If the effect on those who actually use the tool is wanted, divide the ITT estimate by the difference in take-up between the groups. The Wald line of the output shows 1.94 against a true 2.0. That ratio recovers the effect for people who use the tool because they were offered it, and it requires that the offer affects the outcome only through use and that control-group members could not obtain the tool, with the further assumption that nobody was driven away from using it by the offer. Where these fail, the ratio is not interpretable as an effect of use.

ITT does not rescue a poorly randomized or poorly followed pilot. Units must be analyzed in the group they were assigned to even if they leave it, and outcomes should be measured for all of them. A small ITT estimate with low take-up says the offer did little, not that the tool has no value to those who use it. See treatment effect for the quantities involved and selection bias for the general mechanism.

References

• Miguel A. Hernán and James M. Robins, Causal Inference: What If (book and updates)