Model Versioning, Lineage, and Promotion

In this fictional example, a complaints team asks why a customer was declined in March. Version 14 of the credit model had contributed to the decision and had since been replaced twice. The model file was easy to find in the storage bucket. Reconstructing the calculation and the decision around it was harder.

The training table had been rebuilt every night since March, so the data version 14 learned from no longer existed. Two of its features had been redefined in the meantime, and the old definitions survived only in the version history of a shared library that three other models also used. The preprocessing that turned raw fields into model inputs lived in a notebook, and the copy in the repository did not quite match the one that had been run. The approval to put version 14 into production was a message in a team channel. Retraining from what the team believed was the same code produced a model that scored the March customer differently.

Some inputs had disappeared; other components survived without a reliable link to the execution that used them. Three questions now needed different evidence: how version 14 was trained, what it computed for this request, and how that output led to the decline. Versioned lineage helps answer the first. Replaying the request also needs the fitted preprocessing, actual inputs, inference environment, and decision rule. Explaining the business decision requires its recorded reasons and any additional policy or human actions; lineage alone does not establish that the decision was justified.

Four things that change on their own schedules

Artifact	What identifies a version	What breaks without one
Data	An immutable snapshot, a manifest of files, or a table state at a recorded point	The training set cannot be recovered once the source is overwritten or corrected
Features	Definition version, event and availability times, retained value versions or the actual served values	Historical reconstruction may use a value that the service did not have, or silently substitute a new definition
Experiments	A run record linking code version, parameters, data versions, and results	The reason one candidate was chosen over another cannot be reconstructed
Models	An exact artifact identity linked to fitted preprocessing, runtime requirements, and the decision rule	A name or alias alone cannot identify the version that actually served a request or establish recovery compatibility

Google Cloud’s MLOps architecture guidance recommends retaining pipeline and component versions, parameters, output references, and evaluation results for each execution. These links support lineage and investigation; their presence does not prove that inputs remain readable or that an inference can be replayed. Follow a model version back to the relevant data and run, then check the retained evidence. Data Lineage develops the broader idea of recorded derivation relationships.

Data: keep what the model saw

One failure is the one in the opening: a training table rebuilt in place. The query survives; its result does not. Four mechanisms name a data version — immutable snapshots, manifests listing exact files, storage-level object versions, and table formats that can be read as of an earlier state — and each has limits, compared in data versioning. The underlying ideas are covered in snapshot, dataset publication manifest, and object version and ETag.

Record exact input identifiers with training and evaluation runs, and define retention with the applicable data policy and obligations. Where permitted retention cannot support later reconstruction, record that limit rather than claiming full reproducibility. The transformation code and its dependencies also need versioning, as developed in Transformation as Code.

Features: one definition, read as of the right time

A feature is an input attribute or representation; its definition specifies how a value is obtained, as described in feature. Historical reconstruction must distinguish when a source event occurred from when its value became usable. A purchase occurring at noon but arriving at 14:00 cannot inform a 13:00 prediction. A point-in-time join can select eligible historical records, but an event-time cutoff alone may include later corrections or backfills. Preserve definition and value versions, availability information, and enough evidence of caching, expiry, and fallback behavior to support the intended replay.

A feature store supports shared definitions and access to historical and serving values. It can reduce duplicated logic, but upstream computation, materialization, and retrieval still need compatible contracts and checks. Compare served values with reconstructed values for the same entity, request time, and definition version. Feature Store explains the component’s responsibilities and limits; Training-Serving Skew and Feature Stores develops the failure modes and replay checks.

Experiments: record the run, and know what reproducible means

A training run record connects code and data references, settings, results, and produced files; see experiment tracking. Capture evidence from the actual execution. It may be emitted during the run or collected later from its retained logs and artifacts. In either case, preserve run identity, collection scope, provenance, and missing information. Automatic capture can be incomplete or wrong; a later account based only on memory is different from importing execution evidence.

Choose what reproduction must establish. Exact output bytes, numerical agreement within a tolerance, stable individual decisions or rankings, and replication of a statistical conclusion are different targets. Reproducible Build addresses specified output bytes under declared conditions. Reproducible Pipelines covers numerical variation, stable ordering, fitted pipelines, and execution records. Two models can have the same aggregate metric while deciding differently for individual people. Define tolerances and statistical criteria before comparing runs; an observed range across a few reruns is not a guarantee for future runs. Parallel arithmetic and randomness can affect results, but neither makes every execution nondeterministic. The opening’s changed retraining score alone does not identify which difference caused it.

Models: a version is more than a weights file

A usable release needs its learned parameters, fitted preprocessing, configuration, and compatible runtime dependencies. Model Artifact distinguishes these components. They may be packaged together or linked by exact versioned references. To replay a past prediction, load the retained fitted components and the recorded inputs under the required conditions; retraining from the original training data is a separate reconstruction task.

MLflow’s registry associates registered versions with producing runs and metadata, and supports mutable aliases. An alias identifies what a lookup selects at that time; record the resolved version and the version actually serving each request. Model Registry explains why changing an alias does not necessarily reload a running process. Intended uses, evaluation conditions, and known limitations belong in the model card.

Promotion: authorize a candidate, then verify the transition

Promotion authorizes an evaluated candidate for a production role. Preserve the candidate’s identity: an unverified retraining or rebuild does not establish that the result is the model that passed evaluation. Deployment may still create resources and apply environment-specific configuration. An MLflow alias can select a version for a subsequent load, but a service that already loaded a model needs a reload or deployment mechanism. Verify traffic routing and the versions actually responding. Environment Promotion and You Tested a Different Binary develop the corresponding software release discipline.

Set acceptance criteria before examining the final results. Input checks supply evidence within their coverage, not a proof that the feed is correct. Compare the candidate with relevant baselines, including the incumbent, using appropriate uncertainty methods. Specify useful improvement or permitted noninferiority when lower cost, latency, or other requirements motivate the change. Important slices need predeclared limits rather than a demand that every observed number improve; record these in the evaluation protocol.

Revalidate the decision threshold and other decision rules; an unchanged threshold can remain appropriate. If retuning is needed, use development data and then evaluate the fixed rule.

Finally, require independent human approval where policy demands it, or execute a previously approved automatic policy within its scope. Protect the checks, evidence, credentials, and bypass paths as part of separation of duties.

Design recovery around the time and risk the service can tolerate. Keep a usable prior artifact and its preprocessing, runtime, decision rule, and feature contracts. A changed feature definition need not invalidate the fallback if the old version remains available and compatible; silently overwriting its inputs can. Moving an alias back does not itself confirm that every instance has reloaded. Verify the restored service, distinguish that from undoing past decisions, and provide a way to stop further harmful actions while recovery proceeds. Model Promotion develops these checks and their records.

Where most teams start, and what to automate first

Google Cloud’s level-0 example describes manual experimentation and handoff to deployment engineers. Manual work can be a reasonable starting point, but automating an unrecorded process does not make its outputs traceable. Decide what evidence and checks are needed before increasing the rate at which candidates are produced.

Start by connecting execution evidence to each run, including data, code, configuration, results, and artifacts. Retain required input and feature versions under the data policy. Make validation failures block the transitions they are intended to control. Automate repeatable steps within an approved scope; retain a separate human approval wherever it is required. A trustworthy automatic gate needs protected rules and evidence, not just a green status.

The reconstruction test

A practical reconstruction exercise is to select a past prediction and answer the following from retained, authorized evidence. Record gaps as well as successful lookups:

• Which request and timestamp are being investigated, which artifact and fitted preprocessing actually served it, and which runtime, configuration, and decision rules were active?
• Which dataset versions and split records supported training and evaluation, and what can still be read under the retention policy?
• Which input values were actually used, or what evidence can reconstruct them, including definition versions, availability times, cache state, and fallbacks?
• Which run produced the artifact, with what code, configuration, environment, and evaluation evidence? What score and downstream decision were recorded for this request?
• Which approval or authorized automatic policy permitted deployment, what versions actually received traffic, and what recovery path was available?

A registry entry establishes a recorded identity, not the completeness of its lineage or deployment. A dashboard needs traceable evaluation evidence regardless of how entries arrived. A rerun of current code and data addresses the past only if their identities and relevant execution conditions match the historical claim. Evaluate the evidence against the specific question being asked.

In the fictional complaint, the team could assemble only an approximate account from surviving source tables and library history. Missing request-time evidence limited what it could claim. For a later investigation, retain links to training provenance, the fitted model, actual inference inputs and outputs, decision rules, and approval records. The answer’s precision and the time needed to assemble it depend on whether those records cover the request and remain readable, rather than on the number of references alone.

Whether the evaluation evidence supports the model’s intended use is addressed in Ninety-Seven on the Test Set.

References:

• Google Cloud Architecture Center, MLOps: Continuous delivery and automation pipelines in machine learning (Creative Commons Attribution 4.0) — metadata management, the feature store component, model validation, rollback pointers, and the manual starting level.
• MLflow documentation, MLflow Model Registry — lineage, versioning, and aliases in one widely used implementation.
• MLflow documentation, Model Registry Workflows — resolving and loading aliases.
• Feast documentation, Point-in-time joins — reproducing past feature values in one widely used open-source implementation.