Prediction Logging

Prediction logging records evidence of model use: the input values or permitted references, the version that served them, the output, and links to subsequent decisions. Its collection policy determines whether it covers every required request or a sample. It complements experiment tracking by recording what happened on the serving path.

Google’s Rules of Machine Learning recommends retaining served features to compare training and serving calculations. Rule 29 permits a sample for this consistency check. That supports examining the recorded cases; it does not provide an explanation for an unrecorded request.

Link the request to the calculation and the action

1. A stable request identifier and identifiers for related decisions and downstream operations.
2. Prediction and decision times, plus input cutoff or value-age information relevant to validity.
3. The actual model artifact, fitted preprocessing, feature definitions, runtime, and decision-rule versions.
4. Permitted evidence of the inputs actually used, including defaults, missing values, and failed lookups.
5. The raw output — score or probability — before any threshold.
6. The threshold or policy in effect, the resulting decision, recorded reasons, and any later human override or execution outcome.
7. Whether the answer came from the model, a cache, or a fallback.
8. Relevant step latency, timeout status, collection policy, and whether detailed fields were sampled or lost.

What the evidence can establish

Compare actual served features with training-path calculations for matching cases, times, and definition versions to investigate training-serving skew. Google’s Rules of Machine Learning warns that later joins can read values changed since prediction time. The logged inputs help test this mismatch, but do not establish that the source values or shared definitions were correct.

Replaying an inference requires the retained fitted components, runtime conditions, rules, and input evidence. Following the final business decision also requires related policy actions, human judgments, and outcomes. A score and threshold may explain a simple automated branch; they do not by themselves explain or justify every eventual customer outcome. The collection coverage and integrity concerns are those of an audit log. Record what cannot be reconstructed after permitted deletion or collection loss.

Collection, monitoring, and training

Choose minimum decision metadata and detailed payload retention separately, according to the investigation need and data policy. Where every decision must remain traceable, define the required records and reconcile collection gaps. Detailed features may be sampled. Preserve inclusion probabilities and sampling-policy versions, and monitor loss separately from intentional sampling; full request counters can supply population denominators.

Do not treat an unweighted sampled log as the full population. In an illustrative 10,000-request interval with 100 fallbacks, logging all fallbacks and 99 of the 9,900 normal requests produces a log fallback share of about 50.25%, while the actual share is 1%. Known sampling probabilities can support weighted estimates with suitable uncertainty; unknown or outcome-dependent collection loss needs separate investigation. A sample also cannot reconstruct a particular request that it never retained.

Payloads may contain personal data. Set access, retention, and deletion rules as for other data versioning evidence, and preserve the resulting limits on investigation. Joined outcomes can support monitoring and training only for the cases and outcome windows actually observed. Some rejected cases have independent labels; others do not. Training on the observed subset alone can reflect the policy’s selection rather than the intended population.

Logging a random sample does not make missing outcomes appear. Google’s Rule 34 describes a filtering example in which a randomly selected holdout is actually shown to users without the filter, allowing outcomes to be collected. Applying that idea elsewhere requires an approved exploration policy: define which actions may vary, their assignment probabilities, outcome windows, and safeguards. It is not a general instruction to approve suspected fraud. Independent review labels may offer another route, but may measure a different outcome. State which effects remain unidentified when the necessary outcomes cannot be observed.

Use the definition and value-access contracts described in feature store, and record the versions actually loaded rather than inferring them from an alias. Model promotion connects the authorized release to deployment evidence. The Score Was From Tuesday places logging alongside deadlines, fallbacks, and recovery.

References:

• Google for Developers, Rules of Machine Learning — Rules 29–31 on served features and sampling, and Rule 34 on filtering and observed labels (CC BY 4.0)