Model KPIs (Key Performance Indicators)

Model KPIs are the metrics used to judge how a machine learning model performs, how reliably it runs, and what it is worth. They split naturally into technical performance — loss, accuracy, AUC, calibration — and business impact, such as revenue uplift or fraud prevented.

Four families cover the ground. Prediction quality: accuracy, precision, recall and F1, ROC-AUC and PR-AUC, log loss and calibration for classifiers; MSE, RMSE, MAE and \(R^2\) for regressors. Drift and stability: feature drift by population stability index or a KS test, data quality checks, representation drift. Operational: latency, throughput, uptime, cost per prediction. Business impact: incremental revenue, churn reduction, fraud loss avoided, return on the deployment.

A fraud model might report an AUC of 0.92, a 50 ms latency and \$1.2m of fraud prevented in a quarter; a recommender an NDCG@10 of 0.65, a sub-100 ms response and an 8% lift in click-through rate. Listing the four families is the easy part. What the list does not say is how they relate — and they relate badly.

Where the families pull against each other

Better AUC, smaller business gains

Start with a fraud model and a review team that can look at 2,000 cases a day. The simulation has 400,000 transactions. Each transaction has a latent score, and the probability of fraud follows a logistic curve in that score: 0.4% when the score is 0, rising steeply with the score, which gives the sample its fraud share. Each prevented fraud is worth \$320.

import numpy as np
from scipy.stats import norm
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 400_000
fraud_rate, loss_per_fraud = 0.004, 320.0
score = norm.ppf(rng.random(n))
logit = np.log(fraud_rate / (1 - fraud_rate)) + 2.2 * score
is_fraud = rng.random(n) < 1 / (1 + np.exp(-logit))
print(f"{n:,} transactions, {is_fraud.sum():,} of them fraudulent")

400,000 transactions, 10,795 of them fraudulent

A model is a noisy copy of the latent score. Its separation sets how closely it tracks the score, and the team reviews the 2,000 highest-scored transactions. The technical KPI is the model’s AUC; the business KPI is the loss prevented by the frauds the team actually catches.

def frauds_caught(separation, capacity):
    noise = norm.ppf(rng.random(n))
    s = separation * score + np.sqrt(1 - separation ** 2) * noise
    flagged = s >= np.quantile(s, 1 - capacity / n)
    return roc_auc_score(is_fraud, s), (flagged & is_fraud).sum()

previous = None
for separation in (0.55, 0.70, 0.85, 0.95, 0.99):
    auc, caught = frauds_caught(separation, 2_000)
    line = (f"separation {separation:.2f}: AUC {auc:.4f}, {caught:,} frauds caught, "
            f"loss prevented {caught * loss_per_fraud:,.0f}")
    if previous is not None:
        line += f" (+{(caught - previous) * loss_per_fraud:,.0f} over the previous row)"
    print(line)
    previous = caught

separation 0.55: AUC 0.7550, 450 frauds caught, loss prevented 144,000
separation 0.70: AUC 0.8200, 668 frauds caught, loss prevented 213,760 (+69,760 over the previous row)
separation 0.85: AUC 0.8693, 956 frauds caught, loss prevented 305,920 (+92,160 over the previous row)
separation 0.95: AUC 0.9027, 1,218 frauds caught, loss prevented 389,760 (+83,840 over the previous row)
separation 0.99: AUC 0.9145, 1,332 frauds caught, loss prevented 426,240 (+36,480 over the previous row)

The first reading is that technical gains do not convert to business gains at a constant rate. With a review team that can look at a fixed number of cases, AUC climbing from 0.7550 to 0.9145 produces unequal gains, with smaller gains in the later steps — the last step adds 0.0118 of AUC and \$36,480 where an earlier step added \$92,160. The example shows diminishing returns, not a general ceiling imposed by fixed capacity. Better ranking within the top-K selected cases can still improve precision and value at unchanged capacity; global AUC alone does not determine that improvement.

Serving cost against value added

Operational KPIs enter through cost. Three candidate models differ in latency, cost per 1,000 calls and the uplift they add. At 90,000,000 calls a month, the monthly serving cost is the number of thousands of calls times the cost per 1,000 calls, and the value added is calls times uplift times a hypothetical \$0.04, where uplift is the share of calls whose outcome improves. The last column is value added minus serving cost.

Model	Latency	Cost per 1,000 calls	Uplift	Monthly serving cost	Monthly value added	Value minus cost
small, cached	8 ms	0.02	0.30%	1,800	10,800	+9,000
mid-size	45 ms	0.35	0.60%	31,500	21,600	−9,900
large ensemble	260 ms	2.80	0.75%	252,000	27,000	−225,000

The second reading is that operational KPIs are not hygiene metrics beside the accuracy number — at volume they are the decision. Only the smallest model clears its own serving cost; the mid-size one loses \$9,900 a month and the large ensemble \$225,000, and the ensemble is the one that adds the most uplift. Cost per prediction multiplied by call volume is a real number that belongs in the same table as the accuracy, not in an appendix.

One model, several operating points

Last, fix one model at separation 0.85 and change only the number of transactions flagged per day. The scores are a fresh noise draw at that separation, so the 2,000-flag row catches 948 frauds rather than the 956 in the earlier table. Precision is the share of flagged transactions that are fraud, recall is the share of all fraud that is flagged, and F1 is their harmonic mean.

s = 0.85 * score + np.sqrt(1 - 0.85 ** 2) * norm.ppf(rng.random(n))
frauds = is_fraud.sum()

for flagged_per_day in (500, 2_000, 8_000, 30_000, 100_000):
    flagged = s >= np.quantile(s, 1 - flagged_per_day / n)
    tp = (flagged & is_fraud).sum()
    precision, recall = tp / flagged.sum(), tp / frauds
    f1 = 2 * precision * recall / (precision + recall)
    print(f"flag {flagged_per_day:>7,}: precision {precision:.4f}, recall {recall:.4f}, "
          f"F1 {f1:.4f}, frauds caught {tp:,}")

flag     500: precision 0.6200, recall 0.0287, F1 0.0549, frauds caught 310
flag   2,000: precision 0.4740, recall 0.0878, F1 0.1482, frauds caught 948
flag   8,000: precision 0.3285, recall 0.2434, F1 0.2796, frauds caught 2,628
flag  30,000: precision 0.1838, recall 0.5107, F1 0.2703, frauds caught 5,513
flag 100,000: precision 0.0880, recall 0.8155, F1 0.1589, frauds caught 8,803

The third reading is that several of these metrics are the same model read at different thresholds. Across five operating points precision falls from 0.6200 to 0.0880 and recall rises from 0.0287 to 0.8155, with the model unchanged throughout. A threshold change can meet some precision or recall targets by trading off the other quantity, but cannot meet arbitrary targets outside the score distribution’s feasible operating points. A KPI set has to name the operating point, or it is an instruction to move it.

Assembling a set that works

The four families map onto two jobs. Drift and data quality are leading indicators that warn about what may be coming; prediction quality and business impact are lagging indicators that confirm what happened. Operational metrics are service outcomes and constraints; they can also lead later business harm or lag an infrastructure failure, depending on the outcome being monitored, and they bind hardest where the call volume is largest.

Finally, the business KPI is the one that justifies the model and the worst one for detecting that it broke, since it moves for many reasons besides the model. The technical KPIs detect breakage, the business KPIs justify the investment, and using either for the other’s job is how a model comes to be defended with a number that cannot support it.

References

• SciPy documentation, scipy.stats.norm. docs.scipy.org