Decision Flip Rate

The decision flip rate is the fraction of cases for which a decision made by a model or rule differs when it is computed from the new version of its inputs instead of the old one. A model that thresholds a score at 0.5 and sends a message to every customer above it makes a decision per customer; if 16 of 1,500 customers receive a different decision after a pipeline change, the flip rate is 1.07%. The measure applies after the model, so it tracks what users experience, and it is useful as a gate because it compares the two pipelines through the consumer’s own logic.

Why it differs from the input mismatch rate

A new pipeline can change the value of many features without changing many decisions: when the changes are small, only cases near the decision boundary can cross it, while a large change can carry even a case far from the boundary to the other side. The hypothetical figures below illustrate a population of 100,000 scores with standard deviation 1.2 and the decision “score above 0”. The first three scenarios perturb every score by random noise of increasing size. They illustrate how many input changes can coexist with few decision changes when few cases lie near the boundary.

share flagged 0.499
noise sd 0.01  rows with changed score 1.00  flip rate 0.0024
noise sd 0.05  rows with changed score 1.00  flip rate 0.0128
noise sd 0.2   rows with changed score 1.00  flip rate 0.0532
share within 0.05 of boundary 0.0321
systematic shift on 2% rows: flip rate 0.0046

The last line shows the opposite case. In this hypothetical scenario, shifting 2% of the rows by a fixed amount, as a missing feature or a changed default might, flips 0.46% of decisions. A comparison of feature values by column would flag the noisy case as 100% different and the systematic case as 2% different, which reverses the decision-impact ranking for the smallest-noise scenario (0.24% flips versus 0.46%), though the larger-noise scenarios cause more flips. The flip rate ranks them by what each does to the consumer. It also hides things: a defect that moves 20% of one feature by a small amount can leave the flip rate near zero while the stored feature is wrong. For this reason it works as one gate beside contract and column checks, not as a replacement for them.

Setting the limit and reading small samples

• The consumer owner sets the limit. The cost of a wrong decision differs by case. A reminder sent in error may be cheap and a missed one may cost an opportunity. To give the owner evidence, perturb the old inputs by known amounts and measure how the flip rate responds, as illustrated by the scenarios above.
• Use two limits when the gate continues after cutover. A lower limit holds further promotion and a higher one triggers rollback.
• Express small-population limits as counts. With 150 customers in a canary group, one flipped customer is 0.67% and two are 1.33%, so a 0.5% limit means that a single flip causes a hold. State the limit as a number of customers, not a percentage copied from the full population.
• Compare on the same cases. The legacy and new decisions must be computed for the same customers and as-of dates from the same inputs, otherwise population changes appear as flips.

A flip rate depends on the model’s margin distribution. The same pipeline change on a model with many cases close to the boundary gives a higher rate than on a model whose scores are well separated, so a value measured for one consumer does not transfer to another.

References

• Google: Rules of Machine Learning (training-serving skew)