ML Problem Framing

ML problem framing connects a business need to a task a model can learn and a way to judge whether using it helps. Google’s course starts with assessing whether ML is appropriate, then defining the intended outcome, model output, and success criteria.

These choices depend on one another. An output can be predictable yet irrelevant to the available action, or useful in principle but unsupported by accessible data. Revisit the specification as those limits become clear.

A prediction is not yet a decision

A probability estimate does not choose an action. The scikit-learn threshold guide separates those steps: the chance of rain informs whether to carry an umbrella, but the decision also depends on the consequences of carrying one or getting wet. Framing establishes that connection for the application.

Define the decision and test its feasibility

1. Which decision will change? Named, with its owner. If no decision changes, the model is a report. The general discipline is described in decision question.
2. What action follows each prediction? A call, a hold, a price, an order quantity. The action determines what is worth predicting.
3. What exactly is predicted, and when? The target, the population it applies to, and the moment the prediction is made, which fixes what information is legitimately available.
4. How many actions can be taken? A team with capacity for up to two thousand calls needs a priority rule and a final selection, including tie handling. Capacity is an upper bound; cases with negative expected net value need not be selected.
5. How will success be judged, and when? In the business’s terms, over a stated period.

Success metrics are not evaluation metrics

Evaluation metrics measure predictions against labels. Success metrics measure the intended service or business result, as Google’s course distinguishes. A useful evaluation compares deployment with a credible alternative and considers uncertainty, costs, and guardrails over a stated window. An outcome changing after deployment does not by itself identify the system’s contribution. Check the link between a predictive metric and the decision’s goal, following the caution in proxy metric.

How the target can miss the decision

Outcome instead of response. Predicting who will cancel ranks customers by risk; the retention action only helps those whose behavior the action can change. For retention targeting, estimate the action’s incremental effect and combine it with outcome value, costs, and constraints. Conditional average effects are what uplift models estimate.

A convenient label. Clicks stand in for satisfaction, tickets closed stand in for problems solved. A proxy label is sometimes unavoidable, and the framing should record why it is believed to move with the real outcome.

The wrong moment. A model that predicts default using information available only after the loan decision cannot be used at the loan decision. The prediction time is part of the target definition, and getting it wrong produces a model that evaluates well and cannot run.

Signs the framing is wrong

Revisit the framing when predictive metrics improve without credible evidence of business benefit, when users cannot act on the ranking, or when high-risk cases offer little opportunity to help. Unknown error costs limit an economic assessment, but do not prove that a threshold chosen against explicit capacity or policy constraints was arbitrary. Misclassification cost covers consequences, decision threshold covers operating rules, and Machine Learning Fundamentals develops task types and baselines.

How framing, baselines, evaluation, and error costs fit together for a leader deciding whether a model is worth building is worked through in Predicting the Wrong Thing Well.

References

• Google for Developers, Problem framing (CC BY 4.0)
• Google for Developers, Framing an ML problem (CC BY 4.0)
• scikit-learn User Guide, Tuning the decision threshold for class prediction — the separation between the statistical problem and the decision problem.