Data Versioning
Data versioning is the practice of recording exactly which data a process used, in a form that lets the same data be found again later. For machine learning it answers a question that comes up in every audit, incident review, and retraining: which rows, as of when, did this model learn from and get evaluated on?
Google Cloud’s MLOps architecture guidance recommends linking each pipeline execution to prepared data and other outputs, as well as configuration and results. A location alone is insufficient if its contents can change: retain an exact version reference and verify that the referenced data remain available within the retention policy.
Four ways to identify a version
| Approach | How a version is named | Where it breaks |
|---|---|---|
| Immutable snapshot | A copy written once to a dated location and never modified | Storage cost grows with every copy; the discipline fails the first time someone fixes a row in place |
| Manifest | A list of the exact files, with their versions and checksums, that make up one dataset | Only as good as the files it lists staying available; see dataset publication manifest |
| Object versions | Storage-level version identifiers for each file | Identifies files, not datasets; a table spanning thousands of files needs a manifest on top. See object version and ETag |
| Table history | A table format’s record of past states, queryable as of a version or time | History is readable only while required metadata and files remain; expiration or cleanup can make an old reference unreadable |
All four are ways of making a snapshot addressable. The choice matters less than recording the identifier with the run that used it, so that a model version leads to its data version without anyone having to remember.
The overwritten table
A training table rebuilt in place can lose the state used for an earlier model. The query may survive while upstream rows are corrected, late records arrive, or records are deleted. Re-running it can produce a different dataset, although changes outside its selection need not affect the result, and changed inputs need not change every fitted model. The problem is being unable to establish or recover what was used. Keep the prepared result or preserve the inputs and computation conditions needed to reconstruct it.
Retention against deletion duties
Keeping every version forever conflicts with obligations to delete personal data and with storage budgets. Define retention for training and evaluation data under the applicable obligations and policy, including retained versions. Record when authorized deletion or expiration limits reconstruction; the desire to explain a model later does not by itself authorize indefinite retention. Where the data comes from, and what depends on it, is the subject of data lineage.
The minimum record
For data traceability, record input version identifiers, transformation code, time ranges and filters, and the exact prepared-data reference for each run. Reconstructing training additionally requires relevant reference data, resolved settings, runtime dependencies, and split or stable row identifiers; a seed alone cannot restore a split after row order changes. Verify reconstructed artifacts against the chosen reproducibility target. Feature Store addresses feature consistency, while Nobody Could Rebuild Version 14 connects these records to model deployment and prediction replay.
References:
- Google Cloud Architecture Center, MLOps: Continuous delivery and automation pipelines in machine learning (Creative Commons Attribution 4.0) — the metadata recorded for each pipeline run.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.