Model KPIs (Key Performance Indicators)
Definition
Model KPIs are the key metrics used to evaluate the performance, reliability, and impact of a machine learning model.
- They measure how well the model works on prediction tasks.
- Often split into technical performance (loss, accuracy, AUC, calibration) and business impact (ROI, churn reduction, revenue uplift).
Types of Model KPIs
1. Prediction Quality Metrics
- Classification Models
- Accuracy
- Precision, Recall, F1-score
- ROC-AUC, PR-AUC
- Log Loss / Cross-Entropy
- Calibration (are probabilities well aligned with true outcomes?)
- Regression Models
- MSE (Mean Squared Error), RMSE (Root MSE)
- MAE (Mean Absolute Error)
- R² (Coefficient of Determination)
2. Drift & Stability Metrics
- Feature drift (Population Stability Index, KS test).
- Data quality metrics (missing values, schema errors).
- Representation drift (embeddings shift).
3. Operational Metrics
- Latency (time per prediction).
- Throughput (predictions per second).
- Uptime / availability.
- Cost per prediction (compute efficiency).
4. Business Impact Metrics
- Revenue uplift / incremental sales.
- Customer churn reduction.
- Fraud loss savings.
- ROI of model deployment.
Example by Context
- Fraud Detection Model
- Technical KPI: AUC = 0.92
- Operational KPI: 50 ms latency
- Business KPI: $1.2M fraud prevented in last quarter
- Recommendation Model
- Technical KPI: NDCG@10 = 0.65
- Operational KPI: <100 ms response time
- Business KPI: 8% increase in click-through rate (CTR)
Summary
Model KPIs = key metrics that track how well a model performs technically, operationally, and in business terms.
- Leading indicators (like drift) warn about future issues.
- Lagging indicators (like AUC, calibration, loss) confirm actual model impact.