AUUC (Area Under the Uplift Curve)

AUUC, the area under the uplift curve, summarizes a whole uplift curve as one number. The idea is the same as any area-under-a-curve metric: a model that collects incremental response early in the ranking traces a curve that sits high across its whole width, so the area is large. The difficulty is that the phrase names an area without saying what it is divided by, and two widely used libraries divide by different things. On the same model and the same data one reports 0.8873 and the other 0.0212.

What is being integrated

The underlying curve plots cumulative uplift against targeting depth. At a depth of \(n\) customers, with \(n_t\) treated and \(n_c\) controls among them and \(n_{t1}\), \(n_{c1}\) responders in each, the height is \(\left(n_{t1}/n_t - n_{c1}/n_c\right) \cdot n\): the difference in response rates, scaled by the number targeted. Sweeping \(n\) from zero to the whole population traces the curve, and the area under it is the raw AUUC.

That raw area has two properties that make it awkward as a score. It carries units — incremental responses multiplied by customers — and it grows roughly with the square of the population, because both axes grow with \(N\). Radcliffe and Surry handle the same problem in their unscaled Qini measure by dividing the area by \(N^2\) explicitly. Libraries handle it by normalizing against a reference curve, and that is where they part company.

Which AUUC

The data below are synthetic: 20,000 customers, 30 percent of them treated, a base response rate of 5 percent, and a real effect of 6 percentage points confined to the top third of the ranking. Each convention is scored twice, once on the true ordering and once on random scores that carry no information at all.

import numpy as np

def make_population(seed=7, N=20000):
    """A 5 percent base rate, a 6 percentage point effect in the top third
    of the ranking, and a 30 percent treated share."""
    rng = np.random.default_rng(seed)
    treatment = (rng.random(N) < 0.3).astype(int)
    score = rng.random(N)
    p = 0.05 + treatment * np.where(score > 2/3, 0.06, 0.0)
    return (rng.random(N) < p).astype(int), score, treatment

def uplift_curve(y, uplift, treatment):
    """Cut points are the distinct score values, which is what scikit-uplift 0.5.1 does.
       causalml cumulates over every row and interpolates instead, so this function is a
       stand-in for that one rather than a transcription of it."""
    order = np.argsort(uplift, kind="mergesort")[::-1]
    y, uplift, treatment = y[order], uplift[order], treatment[order]
    cut = np.r_[np.where(np.diff(uplift))[0], y.size - 1]
    n = cut + 1
    n_t = np.maximum(np.cumsum(treatment)[cut], 1)
    n_c = np.maximum(n - np.cumsum(treatment)[cut], 1)
    y_t = np.cumsum(y * treatment)[cut]
    y_c = np.cumsum(y * (1 - treatment))[cut]
    return np.r_[0, n], np.r_[0.0, (y_t / n_t - y_c / n_c) * n]

def area(x, y):
    return float(np.sum(np.diff(x) * (y[1:] + y[:-1]) / 2))

def raw_auuc(y, uplift, treatment):
    """The area itself, in units of incremental responses times customers."""
    return area(*uplift_curve(y, uplift, treatment))

def causalml_auuc(y, uplift, treatment):
    """The shape of causalml auuc_score: mean of the gain curve after scaling it to end
       at 1. Built on the distinct-value curve above rather than causalml's per-row
       cumulation, so it agrees on the continuous scores below and not on tied data."""
    x, gains = uplift_curve(y, uplift, treatment)
    return float(np.mean(gains / abs(gains[-1])))

def sklift_uplift_auc(y, uplift, treatment):
    """The shape of scikit-uplift uplift_auc_score: the model's area above the random
    line, over the perfect ranking's area above the same line. The cut points match
    0.5.1; its handling of a zero denominator is simplified here."""
    x, gains = uplift_curve(y, uplift, treatment)
    control_responders = ((y == 1) & (treatment == 0)).sum()
    treated_non = ((y == 0) & (treatment == 1)).sum()
    summand = y if control_responders > treated_non else treatment
    xp, gp = uplift_curve(y, 2 * (y == treatment) + summand, treatment)
    diagonal = area(np.array([0.0, xp[-1]]), np.array([0.0, gp[-1]]))
    return (area(x, gains) - diagonal) / (area(xp, gp) - diagonal)

y, score, treatment = make_population()
useless = np.random.default_rng(999).random(len(y))
for name, fn in (("raw", raw_auuc), ("causalml", causalml_auuc), ("sklift", sklift_uplift_auc)):
    print(f"{name:9} {fn(y, score, treatment):>14.4f} {fn(y, useless, treatment):>12.4f}")
# raw 6447304.9781 3237468.4836
# causalml 0.8873 0.4456
# sklift 0.0212 -0.0030

y2, score2, treatment2 = make_population(N=40000)
# Doubling the population multiplies the raw area by about four.
print(f"{raw_auuc(y2, score2, treatment2) / raw_auuc(y, score, treatment):.2f}")
print(f"{sklift_uplift_auc(y2, score2, treatment2):.4f}")
# 4.34
# 0.0215

The two library-named functions above are simplified stand-ins that show the shape of each convention, not transcriptions of the packages. The two libraries differ from each other as well as from this code. scikit-uplift 0.5.1 cuts at distinct score values, as here, and sets the ratio to zero where an arm’s denominator is zero; this code’s clamping is a simplification of that. causalml cumulates over every row and interpolates instead, which is where the six-row divergence lies — 0.5 from this function against 0.475 from causalml’s published algorithm. The agreement below is on continuous scores over twenty thousand rows and should not be read as a general equivalence with either package. Use the selected library and version for operational comparisons, including its rules for ties and zero denominators.

Every row is a defensible summary of the same curve, and no two of them can be compared. The raw area assigns a useless model half the score of a good one, because it never subtracts the random baseline; a model that ranks at random still accumulates the campaign’s total incremental response, just proportionally rather than early. causalml scales the curve by its endpoint and averages it, which puts a random ranking near 0.5 when the overall effect is positive. Where the campaign’s total effect is negative the endpoint is negative, the scaled curve ends at \(-1\) and a random ranking sits near \(-0.5\); where it is zero the division is undefined, and near zero the ratio is unstable — its own documentation says so, and warns that testing whether AUUC differs from zero “would reject for essentially every model and say nothing about whether the model beats random”. scikit-uplift subtracts the random line first and divides by what a perfect ranking would have achieved, which puts a useless model near zero and makes negative values meaningful.

The second half of the block shows the scale problem directly. Doubling the population multiplies the raw area by 4.34, close to the factor of four that squaring predicts, while the normalized score moves from 0.0212 to 0.0215. Any claim that one model’s AUUC beats another’s therefore needs the two to have been computed on the same data by the same implementation.

AUUC and the Qini coefficient

The two are often described as a raw measure and its normalized counterpart. That is not the distinction. They normalize different curves: the Qini coefficient is built from the Qini curve, which rescales the control arm by the treated-to-control ratio, while AUUC is built from the uplift curve, which rescales the rates. On the data above, scikit-uplift’s qini_auc_score returns 0.1255 for the same model whose uplift_auc_score is 0.0212 — a factor of six between two normalized areas of two curves that describe the same campaign.

causalml draws the same line for a different reason, recommending its qini_score over auuc_score when the question is whether a model beats random: the Qini version is already the area between the model curve and the random curve, “unlike AUUC, whose random baseline is about 0.5”. Both libraries, in other words, treat the Qini side as the one with a meaningful zero. The 0.5 reference for AUUC applies to campaigns with a positive overall effect; on a campaign that lost money it is not the number to compare against, and the endpoint normalization needs an explicit policy for a zero or near-zero total.

What the number is not

• Not comparable across datasets. Radcliffe and Surry’s note on their own measure applies equally here: values “for the same dataset and outcome are comparable, but comparing qini values across different datasets and outcomes can be less meaningful”. A campaign with a higher base rate produces a bigger area for the same quality of ranking.
• Not a statement about calibration. The area depends only on the ordering the scores induce. A model whose predicted uplifts are all ten times too large draws the identical curve.
• Not a business number. It averages over targeting depths, including depths no campaign will ever use. When the volume is fixed, the height of the curve at that volume — or uplift@k — answers the question the budget is asking.
• Not stable enough to read closely. Every point on the curve is a difference between two group rates, so the area inherits that sampling error. A small difference between two models needs uncertainty assessment that respects their shared evaluation data; the replicate experiment under Qini coefficient shows the spread on a similar design.

A reported AUUC is usable when it carries the library and version that produced it, the dataset it was computed on, and whether it was validated on data the model did not see. Without those, the number is an area of something, and which curve and which denominator are left to the reader.

References

• scikit-uplift 0.5.1 documentation, sklift.metrics.uplift_auc_score
• scikit-uplift 0.5.1 source, sklift/metrics/metrics.py (uplift_curve, perfect_uplift_curve, uplift_auc_score, qini_auc_score)
• causalml source, causalml/metrics/visualize.py (get_cumgain, auuc_score, qini_score)
• Nicholas J. Radcliffe and Patrick D. Surry, Real-World Uplift Modelling with Significance-Based Uplift Trees, Portrait Technical Report TR-2011-1, Stochastic Solutions White Paper, 2011