The Copy You Forgot: Applying Data Policy Across a Real Platform

A customer asks you to delete their data. The request is unambiguous and the policy is written down, so someone runs a delete against the customer table and closes the ticket. Meanwhile the record still exists in the warehouse copy loaded last night, in three analyst extracts, in a dashboard’s cached result, in the search index, in the event log the mobile app writes to, and in every backup taken since the account was created.

Nobody was careless. The policy was about data; the platform is about copies. That gap is what data lifecycle work actually consists of, and it is why a policy document and a compliant platform are separated by a great deal of engineering.

Regulatory examples below use one regime — the EU GDPR and the UK regulator’s guidance on it, as they stood in September 2026 — because quoting a specific text is more useful than generalizing across regimes. The obligations differ in detail elsewhere; check your own.

Four stages, and each one depends on the one before

Lifecycle management is usually drawn as a cycle. It is more useful read as a dependency chain, because failures propagate forward and only forward.

• Classify. Decide what kind of data this is and what rules attach to it. These labels are inputs to the policies downstream, alongside purpose, timing, and exceptions.
• Control access. Grant on the basis of the classification and the stated purpose, not on the basis of who asked.
• Retain. Keep it for as long as the purpose requires and no longer, which means someone has to say how long that is.
• Dispose. Delete or anonymise it, everywhere it went, and be able to show that you did.

Classification helps determine which handling rules apply. It does not answer where a particular person’s records live: that needs a separately maintained subject-to-location map. Automatic handling combines both with approved purposes and policies, retention trigger events, and exceptions such as holds. A default policy can deny access to or quarantine unclassified data while its handling is resolved.

Classification is the load-bearing step, and it fails quietly

Data classification assigns each field or dataset to a category that determines how it must be handled — personal, sensitive, confidential, public. A label is one policy input, not enforcement by itself. It fails in three characteristic ways, and none of them announces itself.

• Too many levels. Five confidentiality tiers with overlapping definitions produce classifiers who guess, and a guess recorded as a category is worse than a blank, because the blank would have been noticed.
• Classified once, at creation. A field that was innocuous when added becomes personal data when joined to another one. A free-text note field is the standard example: nothing about its schema says it contains a customer’s medical history, and it often does.
• Applied to sources only. The source table is labelled and the twelve derived tables are not, so controls end at the first hop and the copies are unlabelled by construction.

The third failure is the one worth engineering against, and the mechanism is lineage: if a derived dataset reads a field classified as personal, the derived dataset inherits that classification unless someone explicitly records why it should not — because the data was aggregated past re-identification, or the field was dropped. Inheritance by default and exceptions by decision is the only version of this that survives a platform where new derived tables appear weekly.

Two techniques change what a derivative is, and the distinction matters legally as well as technically: dynamic masking hides values at query time while the underlying data stays as it is, and anonymisation aims to remove the link to a person from the data itself. Dynamic masking controls what a viewer sees. Static masking instead transforms stored values in a copy, such as a test dataset. Whether that copy is anonymous requires a separate assessment of identification risk; changing values alone does not establish anonymity. Anonymising it, if genuinely achieved, takes the dataset out of scope — which is why claims of anonymisation deserve scrutiny rather than a checkbox.

How long is long enough

The regulation states the principle and declines to state the number. GDPR Article 5(1)(e) names the principle “storage limitation” and requires that personal data be “kept in a form which permits identification of data subjects for no longer than is necessary.” The UK regulator is explicit about what that leaves to you: “The UK GDPR does not set specific time limits for different types of data. This is up to you.”

So the number is a decision, and four kinds of argument bear on it.

Basis	Example	How it behaves
A legal obligation to keep	Tax and accounting records for a statutory number of years	A floor. It overrides a shorter operational preference and it is the easiest to evidence
The purpose the data was collected for	Delivery address needed until the order is complete and the return window closes	A ceiling, derived from purpose limitation — once the purpose is spent, keeping it needs a new justification
Analytical value	Two years of behaviour to model seasonality	The weakest basis and the one asserted most confidently. Aggregates often serve it — but aggregating is not the same as anonymising, and a small group or one joinable to other data can still identify people
Dispute and defence	Records that would be needed if a claim were brought	Legitimate, and prone to becoming “keep everything forever” if nobody bounds it

When they conflict, the resolution order is not a matter of taste: a statutory duty to keep beats an operational wish to delete, and a spent purpose beats an analytical wish to keep. The residual conflicts are real decisions and belong to a named owner, not to whoever configured the pipeline.

The output of those decisions is a retention schedule: per category of data, how long it is kept, on what basis, and what happens at the end. The ICO ties this to documentation duties — “you need to establish and document standard retention periods … wherever possible” — and to a recurring action rather than a one-off: “You should review whether you still need personal data at the end of any standard retention period, and erase or anonymise it.” Note the pairing. Retention ending does not have to mean deletion; anonymising satisfies the principle and keeps the analytical value, which is the trade most organizations should be making more often than they do.

One caution about that trade, because it is the point where a lifecycle decision turns into a legal claim. Aggregating is a step toward anonymity, not a guarantee of it. A group of one is the obvious warning sign — not because a count of one is identifying by definition, but because whether it identifies anyone depends on what the group is defined by, what a reader already knows, and what else it can be joined to. “One customer in this postcode with this product” may single out a person to a colleague who knows the area and reveal nothing to a stranger; and a small group can expose a fact about its members without naming any of them, which is disclosure too. So small counts are a trigger for assessment rather than an automatic verdict, in either direction: a suppression rule on cells below a threshold is a sensible default and not a proof of anonymity.

A table of averages by postcode and age band can likewise be linked back with a little outside knowledge. The GDPR test is not whether re-identification is conceivable but whether a person can be singled out or linked to other information using means reasonably likely to be used — which takes account of things like cost, time, and available technology, and considers what someone other than you could do. So treat anonymisation as an assessment with a recorded conclusion, not a transformation you apply. Until it has been made and written down, keep the controls and the retention rule on the aggregate too — dropping them because the data “is only aggregates” is how a deletion obligation gets missed.

One exception is worth knowing about because it is narrower than people assume: data held only for archiving in the public interest, scientific or historical research, or statistical purposes may be kept indefinitely. “We might analyse it later” is not that exception. Two conditions travel with it and are easy to drop: the data has to be used for that purpose and not quietly reused for operations or marketing, and appropriate safeguards have to be in place — access limited to the people doing that work, and the personal detail minimised to what the purpose needs. A dataset parked in the warehouse under the research label, queryable by everyone, is not covered by it.

Deletion is where the platform argues back

The right exists and is stated plainly: Article 17(1) gives a data subject the right to obtain “erasure of personal data concerning him or her without undue delay.” Executing it is an engineering problem whose difficulty varies by an order of magnitude depending on where the data sits.

Where the data is	What deletion takes
Operational database row	A delete statement. The straightforward case, and the one people generalize from
Immutable files in a lake table format	A logical delete followed by physical file rewriting and cleanup. Delta Lake, for instance, removes unreferenced files only when you run vacuum, whose “default retention threshold for the files is 7 days” — so a deleted row remains on disk until that runs, and running it means “the ability to time travel back to a version older than the retention period is lost”
Derived tables, extracts, caches, search indexes	Re-derivation, or a targeted delete per copy. This is where an unenumerated copy survives
Append-only event logs	Usually a redesign: keep identifiers out of the log, or partition by subject so a whole partition can be dropped. Editing history in place is rarely available
Backups and archives	A policy decision, honestly documented. Most organizations cannot surgically edit a backup, so the defensible position has four parts rather than one: a stated retention window after which the backup expires, the reasoning recorded, the backup kept unusable for anything but recovery in the meantime — no ad hoc restores to answer a question, access limited to the people who run recovery — and the deletion list applied on any restore together with valid hold exceptions, so deleted data is not restored to ordinary use and required evidence remains protected

Two practical consequences follow. First, “deleted” needs a definition in your platform — logically unavailable now, physically gone after N days, gone from backups after M months — and consumers of your privacy notice are entitled to a version of that answer. Second, the design decision that makes all of this tractable is made much earlier: identifiers confined to few places, event payloads carrying a subject key rather than embedded personal data, and derived tables rebuildable rather than hand-maintained. Minimisation is usually argued as an ethical principle; it is also the single largest reduction in deletion cost available.

The regulation is, notably, not naive about propagation. Where a controller has made personal data public and is obliged to erase it, Article 17(2) requires reasonable steps — including technical measures, and taking account of available technology and the cost of implementation — to inform other controllers processing that data that the data subject has requested erasure of links to, or copies of, it. Note the shape of that duty: it is an obligation to take reasonable steps and to inform, judged against what is technically and economically feasible, rather than a guarantee that every copy elsewhere disappears. It is a standard you can meet deliberately and fail by accident, and the difference is whether you had enumerated the copies before someone asked.

Automating the parts that repeat

Some of this genuinely automates. Object stores implement a storage lifecycle policy that transitions or expires objects by age without anyone acting, and that mechanism fits only when its clock matches the approved trigger or expiry date, all records in the object share the disposition conditions, and holds and referenced versions remain protected. A file created before a contract ends cannot implement retention measured from contract closure merely by counting its own age. Otherwise, calculate eligible objects or expiry dates in a separate process.

It is a poor fit for retention whose unit is a person, because age of file and age of a subject’s relationship with you are unrelated. Per-subject deletion needs a different mechanism: a stored map from subject to locations, a job that processes requests against it, and a report of what it touched. Building that map is the work; running it is not.

Access control belongs in the same automated tier, expressed against classifications rather than table names, so that a new table carrying personal data is protected because of what it contains and not because someone remembered to grant carefully. Where the data may physically live is a related and separate constraint — see data residency — and it is worth keeping distinct, because a residency rule is about storage location while a classification rule is about handling.

What you have to be able to show

GDPR Article 5(2) names its own principle “accountability” and requires the controller to “be able to demonstrate compliance” — and Article 25(1) asks for “appropriate technical and organisational measures” implementing the principles rather than intentions to. Demonstrating means producing artefacts, and they sort into three kinds.

• The policy. The classification scheme, the retention schedule, the stated purposes, and the privacy notice that tells people what you told them. Dated and versioned, so a past decision can be judged by the rules in force then.
• The application. Evidence that the policy ran: deletion job outputs with counts, expiry runs, access grants and revocations, and the audit log of who accessed what. A policy with no execution record is an intention.
• The exceptions. Every case where the policy was not applied, with the reason, the approver, and the review date. This is the artefact organizations lack, and the one a reviewer asks for first — because the exceptions are where the risk actually is.

The largest category of exception has a name. A legal hold preserves specified content and versions relevant to litigation, investigation, or regulatory inquiry, overriding ordinary deletion while it applies. What makes it dangerous is its tendency to spread and persist: a hold scoped as “anything related to the dispute” over data nobody has enumerated becomes a hold on everything, and a hold with no owner and no end date becomes permanent. Three rules keep it honest — a stated scope in terms of datasets and date ranges, a named owner, and a review date that arrives whether or not anyone remembers the matter.

Held data is still governed data, which is the part most often missed. Access controls and classification continue to apply. Preservation must cover updates, overwrites, and manual deletion as well as expiry jobs. Operational changes can continue when the required earlier content remains in protected copies or version history, with recovery verified. Data parked in an unmanaged copy “for the lawyers” is a breach waiting for a different audit.

Where to start

A lifecycle programme that begins by classifying every dataset produces a spreadsheet and no capability. Starting from a request produces the capability and reveals the gaps in the right order.

• Take one real deletion request and trace it to every copy, by hand, writing down what you find. The list will be longer than expected and it is the inventory you have been missing.
• Turn that trace into a repeatable job for one data domain, with an output report. You now have evidence, not just a procedure.
• Write the retention schedule for that domain’s categories, with the basis for each period. Disagreements will surface here, which is the point — they were always there, unrecorded.
• Make classification inherit along lineage, so the next derived table is covered without anyone acting.
• Only then broaden. A working mechanism on one domain extends; a complete inventory with no mechanism does not.

The measure of whether any of this works is not the existence of a policy document. It is how long it takes to answer three questions: where does this person’s data live, when will it be deleted, and can you show that the last batch of deletions ran. An organization that can answer those in minutes has a lifecycle capability. One that needs a week of investigation has a deletion policy and a platform that does not implement it.

References: Microsoft Azure Blog, Static Data Masking for Azure SQL Database and SQL Server (2018; static/dynamic distinction); Regulation (EU) 2016/679 (GDPR), EUR-Lex; ICO, Principle (e): Storage limitation; Delta Lake Documentation, Utility commands.