Qini Curve

A Qini curve plots the cumulative incremental response of an uplift model against the share of the population targeted, with customers ordered by uplift score, highest first. Radcliffe and Surry, who named the family, describe it as a generalization of the gains curve behind the gini coefficient, and it is from that lineage that the curve and its summary take their name: “Qini measures are based on the area under the incremental gains curve”, and the curve drawn on such a chart is what they call a qini curve.

What the height means

At a depth where \(n\) customers have been targeted, of whom \(n_t\) were treated with \(n_{t1}\) responders and \(n_c\) were controls with \(n_{c1}\) responders, the Qini curve’s value is

\[ \text{qini}(n) = n_{t1} - n_{c1} \cdot \frac{n_t}{n_c} \]

Read it as the estimated responders the treatment produced, minus the responders an equally sized treated group would have produced with no treatment at all. The control arm supplies that counterfactual, and the factor \(n_t/n_c\) is what makes the two arms comparable in size. Both scikit-uplift and causalml implement exactly this expression, so the shape is not a matter of convention between libraries.

The closely related uplift curve rescales the other way, taking the difference of response rates and multiplying by the number targeted. The two are the same curve up to a factor of \(n/n_t\), the reciprocal of the treated share at that depth, so with 30 percent of customers treated a Qini value of 38.94 in the top decile corresponds to an uplift-curve value of 132.24. The factor is taken from the prefix actually used — here \(n/n_t = 2000/589\) — not from the nominal design ratio. Neither is more correct; a number read off one is simply not comparable with a number read off the other, which is where the two are most easily confused.

One consequence of the definition is worth stating before the baselines. The vertical axis is a count, in units of incremental responses, and where it ends is a property of the data rather than of the model. The analogy with an ROC curve, which is often the first one offered, holds for the idea of scoring an ordering and breaks on the axes: an ROC curve runs between fixed corners of a unit square, while a Qini curve ends wherever the campaign’s total incremental response happens to be.

The three reference lines

A Qini curve on its own says little; it is read against baselines, and there are three of them.

• The random line. A straight diagonal from the origin to the curve’s own endpoint. Targeting at random captures incremental response in proportion to volume, so half the population yields half the total. Every Qini curve and its random line finish at the same height; the question is only how quickly each gets there.
• The optimum with no negative effects. The best curve achievable if the treatment never harms anyone: it rises at its steepest until all the incremental response has been collected, then runs flat. Radcliffe and Surry call this the zero-downlift optimum and scale their reported coefficient by it.
• The optimum allowing negative effects. If the treatment can also suppress responses, a perfect ranking would put every harmed customer at the very bottom, and the curve can climb above the eventual total before coming back down to it. This optimum is much higher, which is why Radcliffe and Surry decline to normalize by it: the theoretical best “is often an order of magnitude or more larger than anything achievable”, so every real model is compressed into a band near zero.

A worked example

The data are synthetic: 20,000 customers, 30 percent of them treated, a base response rate of 5 percent, and a real effect of 6 percentage points confined to the top third of the ranking. Scoring is by the true ordering.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def qini_curve(y, uplift, treatment):
    """Treated responders, minus the responders an equally sized treated group
    would have produced without the treatment."""
    order = np.argsort(uplift, kind="mergesort")[::-1]
    y, uplift, treatment = y[order], uplift[order], treatment[order]
    cut = np.r_[np.where(np.diff(uplift))[0], y.size - 1]
    n = cut + 1
    n_t = np.cumsum(treatment)[cut]
    n_c = np.maximum(n - n_t, 1)
    y_t = np.cumsum(y * treatment)[cut]
    y_c = np.cumsum(y * (1 - treatment))[cut]
    return np.r_[0, n], np.r_[0.0, y_t - y_c * n_t / n_c]

def area(x, y):
    return float(np.sum(np.diff(x) * (y[1:] + y[:-1]) / 2))

y, score, treatment = make_population()
x, gains = qini_curve(y, score, treatment)

# The random line runs from the origin to wherever the model curve ends.
diagonal = area(np.array([0.0, x[-1]]), np.array([0.0, gains[-1]]))
# Optimum if the treatment can also harm people, and if it never can.
x_harm, g_harm = qini_curve(y, y * treatment - y * (1 - treatment), treatment)
top = y[treatment == 1].sum() - treatment.sum() * y[treatment == 0].sum() / (1 - treatment).sum()
x_none, g_none = np.array([0.0, top, len(y)]), np.array([0.0, top, top])

print(f"{gains[-1]:.2f} {g_harm[-1]:.2f} {g_none[-1]:.2f}")
print(f"{area(x, gains) - diagonal:.0f}")
print(f"{area(x_harm, g_harm) - diagonal:.0f}")
print(f"{area(x_none, g_none) - diagonal:.0f}")
# 109.08 109.08 109.08
# 851660
# 6787338
# 1084840

Curve	Ends at	Area above the random line
The model	109.08	851,660
Optimum, no negative effects	109.08	1,084,840
Optimum, negative effects allowed	109.08	6,787,338

All three end at the same 109.08, the campaign’s estimated incremental responses scaled to the treated group’s size, with the control arm supplying the counterfactual; what differs is the path. The two optima differ in area by a factor of 6.26 on this data, and dividing the model’s area by one or the other gives 0.785 or 0.126 for the same model — which is the whole content of the warning under Qini coefficient about reporting which normalization was used.

Reading the shape

A curve that leaves the origin steeply has put customers with real effects at the top of the ranking. A curve that tracks the diagonal has an ordering that carries no information about the effect, whatever the model may know about who responds.

Height at a chosen depth is the number to compare when the campaign volume is fixed, and it does not always favor the same model. In Radcliffe and Surry’s illustration one model delivers an estimated 450 incremental sales at 20 percent volume against another’s 380, and above 40 percent the ordering reverses, so neither model dominates the other. Which is better is a question about the volume you intend to send, and a single summary number over all volumes averages across depths you will never use.

A curve that peaks before the right-hand edge has two possible causes and the chart cannot separate them. One is a genuine negative effect: Radcliffe and Surry show a model reaching more incremental sales at 80 percent volume than at 100, “because of negative effects in the last two deciles”. The other is noise, since every point is a difference between two group rates and the segments at the bottom of a ranking are the ones where that difference is least reliable. The size of the segments and a second curve on held-out data are what settle it.

Where to cut is a question the curve alone does not answer. Applying cost and value converts it into a profit curve: with treatment costing \$1,000 per 1 percent of the population and each incremental sale worth \$150 in contribution, Radcliffe and Surry’s first model peaks at 40 percent volume with 680 incremental sales and about \$62,000, and their second at 60 percent with 870 sales and about \$70,500. Both optima sit well short of the point where the curve stops rising.

What it assumes

The curve compares treated and control customers as they fall in the model’s ordering, and that comparison is causal here because assignment was randomized. On data where the previous campaign chose its own recipients, an unadjusted Qini curve measures the old targeting rule as faithfully as it measures the new model. Observational data is not hopeless — propensity weighting or outcome adjustment under conditional exchangeability and overlap can recover the comparison — but the raw group difference does not, and it looks exactly the same either way — see A/B testing. The curve also depends only on the ordering, so a model whose predicted uplifts are all wrong by a constant factor draws the identical curve as one that is calibrated; that distinction starts to matter the moment a cut-off is chosen from predicted values rather than from a fixed volume.

Finally, every point is a segment estimate and segment uplift estimates are not additive, so per-segment values cannot be added up to a prefix value. The prefix value itself is unchanged by the resolution; what changes is where the curve is sampled and how the area between samples is interpolated. The arithmetic behind that is set out under cumulative uplift.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (qini_curve, perfect_qini_curve, uplift_curve)
• causalml source, causalml/metrics/visualize.py (get_qini, qini_score)