Qini Coefficient

The Qini coefficient summarizes an uplift curve as a single number. Radcliffe and Surry define the family in one line — “Qini measures are based on the area under the incremental gains curve” — and present it as the uplift analog of the gini coefficient, which is itself the area between a gains curve and the diagonal of random targeting, normalized. The name is the same word with the first letter changed, and the analogy is the fastest way to remember what it does: like gini, it scores a ranking.

The analogy is a good guide to the sign and a poor one to the range. A gini coefficient runs from +1, for a model that ranks every responder ahead of every non-responder, through 0 for a model that performs as well overall as random targeting, to -1 for the worst possible ordering. A Qini coefficient shares the reading of the sign — positive means the ranking beat random targeting, negative means it did worse, and a negative value is not a bug but a statement that the wrong customers were put first — but neither of the two normalizations implemented below guarantees a \(\pm 1\) range.

The denominator is computed from a reference curve, but that does not establish a symmetric range: the implementation below returns -1.3222 on an eight-row sample. Values below -1 are therefore reachable; how far below is not something these examples settle. And with negative_effect=False the denominator can be smaller than the area a real model achieves, so the same function returns 6.2565 on another population.

Under that variant a campaign with no overall uplift leaves the zero-downlift curve enclosing no area, and the ratio is undefined — that is a property of that normalization, not of the metric: overall uplift of zero alone does not determine whether the default denominator vanishes. For y=[1,0,1,0], treatment=[1,1,0,0], and score=[4,2,1,3], the default coefficient is 1.0 despite equal arm response rates. Replacing every outcome with zero makes the default denominator zero too. The example function raises ZeroDivisionError for a zero denominator; report that coefficient as undefined rather than interpreting it as a zero-quality ranking. Reversing the ranking of a working model in the example below turns +0.1255 into -0.1253.

What the number is divided by

The numerator is settled: the area between the model’s incremental gains curve and the diagonal. The denominator varies by convention, so a change of convention can change the reported score without changing the ranking.

The obvious denominator, by analogy with gini, is the area under the best curve any model could draw. Radcliffe and Surry decline it for two reasons. The theoretical optimum “is often an order of magnitude or more larger than anything achievable”, which compresses every real model into a narrow band near zero, and it is not well defined for outcomes that are not binary. They scale instead by the zero-downlift optimum, the best curve available on the assumption that the treatment harms nobody, and call the result \(q_0\): “the ratio of the area of the actual incremental gains curve above the diagonal to the zero-downlift incremental gains curve”. Their own caveat follows immediately — if the overall uplift in the data is zero, the zero-downlift curve encloses no area either, and \(q_0\) is infinite.

The alternative they describe as more generally useful is the unscaled measure \(Q\), “the area between the actual incremental gains curve in question and the diagonal corresponding to random targeting”, divided only by \(N^2\) to remove the dependence on population size. This scaling removes the square dependence on sample size; it does not express the score as a fraction of an optimum. Its numerical range depends on the outcome and curve convention. Compare models on the same evaluation population and outcome rather than treating it as a universal percentage of ranking quality.

Software makes its own choice. scikit-uplift’s qini_auc_score is documented as the normalized area under the Qini curve, “the ratio of the actual uplift gains curve above the diagonal to that of the optimum Qini curve”, and computes \((A_\text{model} - A_\text{diag}) / (A_\text{optimum} - A_\text{diag})\). Its negative_effect argument selects which optimum: the default optimum assumes the campaign can also actively harm people and is therefore much higher, while negative_effect=False uses the zero-downlift curve. On the data below the same model scores 0.1255 by default and 0.7851 with the flag turned off. Neither denominator delivers a \(\pm 1\) range. The default normalization returns -1.3222 on the eight-row sample above; that establishes a value below -1, not an unbounded negative range. The zero-downlift variant returns 6.2565 where the observed gain exceeds its reference area, so 1 is not its upper bound. Neither is wrong, and neither carries a \(\pm 1\) range. A Qini coefficient quoted without saying which denominator produced it is uninterpretable, and one quoted with a stated range needs that range checked against the implementation rather than inherited from the gini analogy.

A worked example

The data are the synthetic population from the uplift curve entry: 20,000 customers, 30 percent treated, a 5 percent base rate, and a 6 percentage point effect confined to the top third of the true ranking. Scoring by the true ranking is optimal in expectation rather than an upper bound on any realized curve — the effect is constant within the top third, so orderings inside it are interchangeable in expectation and differ by sampling noise. The implementation below reproduces the core binary-outcome calculation for these examples. It assumes nonempty, equal-length arrays, binary outcomes and treatment indicators, finite scores, and both treatment arms. It is an explanatory implementation, not a replacement for the library’s input validation.

After sorting by predicted uplift, each prefix contains treated responses \(y_t\) and control responses \(y_c\). The gain \(y_t-y_c n_t/n_c\) subtracts the control response count rescaled to the number treated. Trapezoidal integration adds the areas between successive prefixes; subtracting the diagonal compares this area with the reference for random targeting. Equal scores enter together so arbitrary ordering inside a tied group does not determine its reported point. A prefix with no controls has no observed comparison: the code’s denominator clamp is a numerical convention, not evidence of a causal effect.

import numpy as np

def make_population(seed=7, N=20000):
    """20,000 customers, 30 percent treated, a 5 percent base rate,
    and a 6 percentage point effect confined to the top third of the ranking."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def qini_curve(y, uplift, treatment):
    order = np.argsort(uplift, kind="mergesort")[::-1]
    y, uplift, treatment = y[order], uplift[order], treatment[order]
    cut = np.r_[np.where(np.diff(uplift))[0], y.size - 1]
    n = cut + 1
    n_t = np.cumsum(treatment)[cut]
    n_c = np.maximum(n - n_t, 1)
    y_t = np.cumsum(y * treatment)[cut]
    y_c = np.cumsum(y * (1 - treatment))[cut]
    return np.r_[0, n], np.r_[0.0, y_t - y_c * n_t / n_c]

def area(x, y):
    return float(np.sum(np.diff(x) * (y[1:] + y[:-1]) / 2))

def qini_coefficient(y, uplift, treatment, negative_effect=True):
    x, gains = qini_curve(y, uplift, treatment)
    # The default optimum allows the campaign to harm people; the alternative is
    # the zero-downlift optimum, which assumes it never does.
    if negative_effect:
        xp, gp = qini_curve(y, y * treatment - y * (1 - treatment), treatment)
    else:
        top = y[treatment == 1].sum() - treatment.sum() * y[treatment == 0].sum() / (1 - treatment).sum()
        xp, gp = np.array([0.0, top, len(y)]), np.array([0.0, top, top])
    diagonal = area(np.array([0.0, xp[-1]]), np.array([0.0, gp[-1]]))
    return (area(x, gains) - diagonal) / (area(xp, gp) - diagonal)

y, score, treatment = make_population()
print(f"{qini_coefficient(y, score, treatment):.4f}")
print(f"{qini_coefficient(y, score, treatment, negative_effect=False):.4f}")
print(f"{qini_coefficient(y, -score, treatment):.4f}")
# 0.1255
# 0.7851
# -0.1253

def summarize(values):
    return (f"{np.mean(values):+.4f} {np.std(values, ddof=1):.4f} "
            f"{min(values):+.4f} {max(values):+.4f}")

uninformative, real = [], []
for s in range(40):
    y_s, score_s, t_s = make_population(s)
    uninformative.append(qini_coefficient(y_s, np.random.default_rng(1000 + s).random(len(y_s)), t_s))
    real.append(qini_coefficient(y_s, score_s, t_s))
print(summarize(uninformative))
print(summarize(real))
# +0.0071 0.0205 -0.0343 +0.0487
# +0.1112 0.0167 +0.0692 +0.1488

Three numbers for one model on one dataset, and the first two differ only in the optimum curve used as the denominator — a factor of six from a keyword argument. scikit-uplift’s uplift_auc_score, which normalizes the area under the uplift curve rather than the Qini curve, moves it again, to 0.0212 for the same model. This is the practical content of Radcliffe and Surry’s footnote that values “for the same dataset and outcome are comparable, but comparing qini values across different datasets and outcomes can be less meaningful”. Report the library, its version, and the normalization, or the number travels without its meaning.

How much of it is noise

A Qini coefficient is computed from differences of response rates in nested subgroups, and differences are noisier than levels. Running the same generator forty times with different seeds, and scoring each replicate twice — once with the true ranking and once with random scores that carry no information — gives two distributions.

Scoring	Mean	Standard deviation	Range over 40 replicates
Random scores (no information)	+0.0071	0.0205	-0.0343 to +0.0487
True uplift ranking	+0.1112	0.0167	+0.0692 to +0.1488

The uninformative ranking reaches +0.0487 in these forty replicates. A positive value alone therefore does not establish useful targeting. These simulated ranges are not confidence intervals or universal significance thresholds; sample size, response rates, treatment allocation, effect heterogeneity, and normalization all affect uncertainty. Compare candidate rankings on held-out data, and quantify uncertainty using a procedure that respects the experiment’s assignment unit.

Radcliffe and Surry respond to this in two ways that are worth copying. They prefer validated Qini, the value on data the model did not see, when choosing between models. And they deliberately bias their estimates downwards, resampling the training population with replacement to build a distribution of Qini values and then taking a pessimistic point from it, so that a model is not selected for having been lucky. The bootstrap is the general form of that procedure.

What it does not measure

“The qini measure, like gini, is a measure only of the rank ordering performed by the model.” That is enough when a fixed volume has to be treated and the only question is whom to pick. It is not enough when the cut-off is chosen from predicted values, because then the correspondence between a segment’s predicted uplift and its actual uplift matters, and Qini is blind to it.

Radcliffe and Surry add a second limitation that is easy to miss, because it cuts the other way. Accuracy at segment level is not sufficient either: a model that assigns random segments the population average uplift will predict those segments almost perfectly and be useless, since it makes no distinctions to act on. Where effects actually differ, a targeting model should distinguish those differences while keeping segment estimates accurate. If effects are genuinely constant, inventing score variation does not improve targeting. Their working list of what to look at alongside the coefficient includes the validated value, whether incremental gains fall monotonically down the ranking for reasonably large segments, the highest uplift actually reachable where negative effects exist, the performance at the cut-off if one is fixed, and how closely the validation pattern matches the training one.

One further caution concerns using the measure as a fitting objective rather than a report. They tried Qini directly as a tree splitting criterion and had “only limited success”, because a measure that responds only to rank ordering gives a split little to aim at. Their production trees split on a significance-based criterion and use Qini to assess the finished model; causal trees take a different route to the same problem.

Where it fits

Qini is to an uplift model roughly what AUC is to a classifier: a summary of the ordering that says nothing about the predicted values, and is best read next to the curve it came from. The difference is that a classifier’s AUC is computed against labels that exist, while every point behind a Qini coefficient is a difference between two groups, estimated with sampling error. More independent evaluation data can reduce uncertainty in group effects, although it cannot reveal both potential outcomes for one individual. When the decision is which of two models to deploy at a known volume, the height of the uplift curve at that volume answers the question more directly, and a summary over volumes you will never send is a summary of the wrong thing.

References

• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011
• scikit-uplift 0.5.1 documentation, sklift.metrics.qini_auc_score
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (qini_curve, perfect_qini_curve, qini_auc_score)