Batch Scoring
Batch scoring applies a model to a bounded set of cases and prepares predictions for later use. Google’s Machine Learning Crash Course calls this static or offline inference, contrasting it with dynamic inference performed on demand. A batch may run on a schedule or be triggered for a particular input set.
The lookup path retrieves a prepared prediction rather than rerunning that model for the request. It still performs access, validity, and decision checks. The model runs as a batch process over a bounded input; making the published results usable is a serving responsibility.
What it gives, and what it costs
| Gives | Costs |
|---|---|
| Fast retrieval without executing the scoring model in the request path | Validity depends on the input cutoff, calculation and publication times, and the decision horizon |
| A complete result set can be validated before release | Only the cases in that published set are covered; a partial write must not bypass validation |
| More room for an expensive model when the batch completion deadline allows it | Computation and storage may be spent on predictions that are never used |
Online prediction computes new cases when requested, within the supported input and feature contracts. Its inputs may still be delayed or cached, and output validation can precede action. Google’s course notes that latency and computation limit how complex a dynamically served model can be, and being online does not guarantee that every input can be scored within a particular service deadline.
When it fits
Batch scoring fits when the relevant cases can be selected in advance and the predictions remain useful until consumption. A daily call list is one example; an expensive model with a generous completion window is another. Compare refresh cost and the share of predictions actually used against on-demand calculation. Where some cases cannot use a prepared score, define a valid online, rule-based, deferred, or rejection path; absence from the batch is not a negative prediction.
Publish a verified result and preserve its meaning
Keep the model and fitted preprocessing identity, input version or manifest, input cutoff and expected coverage, calculation time, and publication time with the result version. Define the target period and the conditions under which a consumer may use a score. A score calculated an hour ago can still rely on a cutoff 49 hours ago; a successful run does not prove that expected source data arrived.
One release design writes a new immutable result version away from live lookups, validates coverage, nulls, and relevant output distributions, then atomically changes a published pointer. A consumer resolves that pointer once for an operation and reads only that version. Pointer replacement alone does not prevent mixed reads if consumers resolve it again for each row. Keep the prior version available while readers use it, subject to its validity limits.
Retry a failed run into an isolated version or use a retry-safe write contract keyed by run and case. Do not append duplicate results or leave removed cases silently reachable as current scores. A complete replacement set can make absent cases explicitly unscored; a delta design needs explicit removals. If cross-case consistency is unnecessary, a per-case version contract may suffice, but state which mixtures readers are allowed to observe. Validate the same result version that will become visible.
How batch and online paths are chosen and combined, including their latency, cost, and failure behavior, is worked through in The Score Was From Tuesday.
References:
- Google for Developers, Machine Learning Crash Course: Static versus dynamic inference (Creative Commons Attribution 4.0).
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.