Treatment Effect

A treatment effect is the change in an outcome caused by a treatment, compared with what the outcome would have been without it. The potential outcomes framework makes that precise. For one unit, write \(Y(1)\) for the outcome if the unit is treated and \(Y(0)\) for the outcome if it is not; the individual treatment effect is the contrast \(Y(1) - Y(0)\). The difficulty is stated in the standard reference: individual effects are a contrast of counterfactual outcomes, only one of which is ever observed for a given individual, and so “Because of missing data, individual effects cannot be identified”. Each unit is either treated or not, so one of the two outcomes is always missing, and no design fixes that.

What can be estimated is an average. The reference defines it for the population: “An average causal effect of treatment A on outcome Y is present if” the outcome’s expected value differs under the two treatments, which it writes as \(E[Y^{a=1}] \neq E[Y^{a=0}]\), a definition that covers binary and continuous outcomes alike. In experimental language that expectation of the difference, \(E[Y(1) - Y(0)]\), is the average treatment effect, and it is what a randomized trial or an A/B test estimates by comparing group means. The reason randomization works is stated in the same source: “randomization ensures that those missing values occurred by chance”. Assignment by chance makes the two groups exchangeable in expectation, so the difference in their mean outcomes is an unbiased estimate of the average effect; it does not make any particular pair of groups identical, and in a finite sample chance imbalances remain, which is what the standard error accounts for. Randomization also fixes what is being estimated: comparing the groups as assigned estimates the effect of assignment, and when some of those assigned do not take the treatment, that is not the same quantity as the effect of receiving it. The broader definition and the conditions under which observational data can identify it are the subject of the causal effect and causal inference entries.

Which average is meant

These are different numbers, and they can differ in sign. A retention offer might have a positive ATT, because the people who accepted it were the ones it could help, and a near-zero ATE, because most of the population would have stayed anyway. The conditional effect is where heterogeneity lives; the reference’s chapter on the subject is titled “Heterogeneity of treatment effects”, and it is the quantity that uplift models try to predict per unit in order to decide whom to treat. A model that predicts an outcome ranks people by how likely the outcome is, not by how much the treatment changes it, and the two orderings can be almost unrelated.

What an estimate has to state

A treatment effect estimate that is reported as a single number is incomplete until it says what it is an estimate of, because each of the following changes the number. The population: all customers, new customers, the customers who were eligible. The treatment contrast: this offer versus no offer, or versus the previous offer. The outcome: conversion within seven days, revenue within ninety. The scale: an absolute difference or a ratio. And the time and conditions under which the effect was measured, since a different season or a longer window is a different quantity. A trial report that says recovery at 30 days was 60 percent with the drug against 50 percent with placebo, among adults admitted with the condition, has stated the population, the comparison, the outcome, the scale, and the follow-up; an analytics report that says the campaign lifted conversion by 10 percent has usually stated none of them.

The scale deserves its own sentence, because the same trial gives two true numbers. If recovery is 60 percent with the drug and 50 percent without, the effect on the difference scale is 10 percentage points and on the ratio scale is a 20 percent relative increase; a conversion rate that moves from 5.0 to 5.5 percent is half a point absolute and 10 percent relative. The reference is neutral between them, “The use of either the multiplicative or additive scale will depend on the goal of the inference”, and notes a subtlety that matters for reporting: the population difference is the average of the individual differences, but “the causal risk ratio in the population is not the average of the individual causal effects” on the ratio scale. For a binary outcome, an absolute effect multiplied by the number of people gives the expected net change in the number of events, which is not the number of people helped: if the treatment helps some and harms others the net figure hides both, and without a further assumption, such as that no one is harmed, the number of beneficiaries is not identified. Relative effects compare across base rates. Say which scale is meant.

Estimation, and what a method does not supply

In a randomized experiment the difference in group means estimates the average effect, with a standard error and interval computed as for any difference of means, and the main risks are a mismatch between the unit that was randomized and the unit analyzed, which when clusters were assigned calls for a standard error that respects that structure; a stopping rule that was not planned; and a choice of outcome made after the data were seen. Without randomization, the same difference is an association that a set of assumptions may or may not turn into an effect. Matching, regression adjustment, propensity scores, instrumental variables, and difference-in-differences are ways of using the data under those assumptions; none of them supplies the assumptions, and a precise estimate from a method whose assumptions fail is precisely wrong. The causal inference entry sets out the conditions and why the data cannot confirm them.

• Which estimand: ATE, ATT, a conditional effect, or a local one? For whom is the average taken?
• What are the population, the contrast, the outcome, the scale, and the window?
• Was treatment randomized, and at which unit? If clusters were assigned, does the analysis account for that; if nothing was randomized, which assumptions identify the effect, and who checked them?
• Is the reported number absolute or relative, and would the other scale tell a different story?

References

• Miguel A. Hernán and James M. Robins, Causal Inference: What If (Chapman & Hall/CRC; online edition of 2 January 2025), chapters 1, 2, and 4