Access, Retention, and Erasure Across Data Copies

Suppose an organization receives a verified request to erase one customer’s personal data. An engineer deletes the row from the operational database and the application confirms it within seconds. This article is about everything that confirmation does not cover. In the following illustrative platform, the customer’s identifier also sits in a raw change-data-capture topic, a cleaned warehouse table, a feature table used by a model, a search index, a file export sent to a partner, a manually built dashboard extract, a set of nightly backups, and the older versions that the table format itself retains. Each of these is a copy with its own readers, its own lifetime, and its own way of forgetting.

The article treats access, retention, and erasure as one problem seen from three sides. Access asks who may read a copy. Retention asks how long a copy may exist and what happens at the end. Erasure asks how a specific person’s data is removed from every copy before its normal expiry. All three depend on the same two facts: knowing what each piece of data is (classification) and knowing where it went (lineage). The sections below follow that order: classification, authorization, derived copies, deletion propagation, and finally the exceptions that backups introduce and the audit evidence that lets you show what happened.

It builds on the previous article, master data and entity resolution, which is about deciding which records in different systems refer to the same real-world entity. Here we use only one consequence of that work: a stable subject identifier that lets a platform find one person’s records across systems. Where that identifier is unreliable, every technique below degrades, and the text says so. This is an engineering article, not legal advice. The rules that apply to a given organization depend on jurisdiction, legal basis, and contract, and the legal text is summarized only to show what the engineering has to support.

The unit of the problem is the copy, not the table

A data platform exists to make copies. Ingestion copies source data into a landing area, transformation produces cleaned and aggregated tables, serving layers copy rows into caches and search indexes, and people export files. Each copy is made for a legitimate reason, and each inherits the sensitivity of its source without automatically inheriting the controls. The table below groups the copies that commonly appear and the property that makes each one hard to handle. It is a working taxonomy for the rest of the article, not a standard.

Copy type	Example in the illustrative platform	Why it is hard to govern
Replica or landing copy	Raw change-data-capture topic and landing files	Append-only by design; history is the point; replay can re-create rows that were deleted downstream
Transformed copy	Cleaned customer table, joined order table	Same grain as the source, so a subject key usually exists; labels must be inherited, not re-assigned by hand
Aggregated copy	Revenue by month and region	May no longer identify anyone, but only if groups are large enough and the output cannot be linked to other data; that is a judgment, not a rule
Derived representation	Feature table, embeddings, a model trained on the rows	The subject key may be hashed or absent; the information can persist in parameters or vectors
Serving copy	Search index, cache, materialized view	Often refreshed asynchronously; a deletion in the source can be invisible here for a while
Outbound copy	Partner file, spreadsheet extract, notebook output	Leaves the platform; the platform can no longer delete it, only notify the recipient
Version history	Table snapshots, object versions, time travel windows	Deleted rows remain readable in older versions until the retention window ends and cleanup runs
Backup	Nightly snapshot, log archive	Immutable by purpose; individual rows cannot be edited out
Evidence	Access logs, audit events, job logs	Can contain the personal data that the main system has deleted, if payloads were logged

Two properties run through the table. Some copies are meant to be changed (serving copies, current table state) and some are meant not to be (backups, append-only logs, immutable files). Erasure is easy in the first group and requires a design decision in the second. And some copies stay inside the platform’s control while others do not. For the second group, the achievable control is contractual and procedural, not technical.

Classification: labels as metadata the platform can act on

Classification is the act of assigning each dataset and field to categories that policies can consume. It is the first requirement because every later step branches on it: who may read a field, how long it may be kept, whether a deletion request applies to it. A data classification scheme that is a single scale (public, internal, confidential, restricted) tends to collapse three different questions into one. It helps to keep them separate.

Axis	Question it answers	What it drives in the platform
Personal data	Does this relate to an identifiable person, directly or in combination with other data?	Whether erasure and purpose rules apply; the subject key needed to find the data
Confidentiality	What is the damage if the wrong people see it?	Access grants, masking, sharing and export rules
Regulatory or contractual regime	Which named rules apply, such as financial records or payment card data?	Minimum retention, required evidence, restrictions on where data may live

The axes are independent. A quarterly financial forecast can be highly confidential and contain no personal data. A business email address that the owner published for enquiries is personal data with low confidentiality in that public context. GDPR defines personal data as any information relating to an identified or identifiable natural person, which includes identifiers and online identifiers; a field is personal if a person can be identified directly or indirectly, even when the field alone does not name anyone. The consequence for classification is practical: label the personal-data axis by asking what the field could be joined to, not by looking at the column name.

What a usable label looks like

A label that lives in a policy document cannot be enforced. A label that lives in the catalog next to the column can be read by the job that grants access, the job that expires data, and the job that executes erasure. The record below is an illustrative shape for such metadata; the field names are invented for this article and are not tied to a particular catalog product.

dataset: silver.customers
owner: customer-data-product          # a named role, not a person's name
columns:
  - name: customer_id
    personal_data: true               # subject key for erasure and access requests
    confidentiality: internal
    subject_key: true
  - name: email
    personal_data: true
    confidentiality: restricted
  - name: plan
    personal_data: false              # but linkable through customer_id
    confidentiality: internal
retention:
  rule: customer-account            # points to a retention rule, not a number
  clock_starts: account_closed_at   # the trigger event, stored with the data
erasure:
  action: delete_rows               # delete_rows | crypto_erase | aggregate_only | not_applicable
  subject_key_column: customer_id

Three details in the record matter. The subject_key flag is what lets an erasure job find rows for one person. The retention rule names a trigger (account_closed_at), because “keep for three years” is incomplete until you say three years from what. And the erasure action is stated per dataset, because the right action differs: delete the rows in a transformed table, destroy an encryption key in a table where rows cannot be rewritten cheaply, or keep only an aggregate.

How classification fails

The catalog can hold perfect labels and the platform can still leak, because classification fails in recurring ways. Labels are assigned once at creation and never revisited, so a free-text notes column that was innocuous by design ends up holding health details. Labels are applied to sources and not to the twelve tables derived from them, so controls stop at the first hop and the copies people actually query are the unlabelled ones. And labels have no owner, so a classification made by whoever built the pipeline reflects what that person knew, with no one to ask when it looks wrong.

The countermeasure for the second failure is inheritance along lineage. Where the lineage graph is known, a derived column inherits the strictest label of its inputs by default, and lowering a label requires a recorded decision. The reasonable default depends on what the transformation did: a direct copy inherits; a hash or token inherits until someone argues otherwise, because a linkable value still identifies a person; dropping a field permits lowering its label only after assessing whether the remaining columns identify the person and whether membership in the result discloses a sensitive condition used to filter or join the rows. Removing a diagnosis column from a list selected for that diagnosis does not remove what the list reveals. An aggregate needs an assessment. Automated scanners that detect card numbers or email addresses help find candidates, but their results depend on configured detectors and formats, and they miss free text and identifiers that only mean something inside the organization. Treat a scanner result as evidence to validate, not as the classification itself.

Authorization: the same data, a different decision per copy

Once data is classified, authorization decides who may read which part under which purpose. Two features of data platforms make this harder than in an application. First, the decision has to be made at many enforcement points, since the warehouse, the object store, the search service, and the notebook environment each authorize independently. Second, purpose matters: personal data collected to fulfill an order may not be reused for an unrelated analysis simply because an analyst has table access. The purpose limitation principle makes the approved purpose part of the access decision, not only of the privacy notice.

Role-based and attribute-based decisions

The two common ways to write the rule are described in role-based and attribute-based access control. A role-based grant says that members of a role may perform an action on an object. An attribute-based rule evaluates a policy over attributes of the subject, the object, and the context, so it can express “analysts in the fraud-investigation group may read fields labelled restricted, for tables tagged with purpose fraud”. For classified data the attribute model is the natural fit, because the classification is already an attribute of the object. The cost is that the policy is exactly as correct as the attribute it reads. If a column’s label is missing, stale, or inherited from the wrong template, a rule that was meant to deny will permit, and the log will show a decision that looks correct. The rule for missing attributes must therefore be deny, not a default that happens to allow.

Row- and column-level controls apply the decision inside a table. Row-level security restricts which rows a viewer sees, and data masking hides or substitutes values in what is returned. Their weakness is their coverage. A rule enforced in one report protects that report, and a copy, an export, or a direct connection to the warehouse leaves it behind. The pattern below shows the idea in plain SQL using a view that exposes a masked column to most roles. The function that identifies the current role differs between engines, so treat this as a schematic of the structure and not as portable code.

-- Illustrative. current_user_in_role() stands for an engine-specific function.
CREATE VIEW analytics.customers_masked AS
SELECT
  customer_id,
  CASE WHEN current_user_in_role('pii_reader')
       THEN email
       ELSE 'redacted'
  END AS email,
  plan
FROM silver.customers;

-- Base table is not granted to analysts; only the view is.
REVOKE SELECT ON silver.customers FROM analyst;
GRANT  SELECT ON analytics.customers_masked TO analyst;

The code shows where the enforcement boundary sits: analysts never receive a grant on the base table, so the only path to the email column goes through the view that evaluates their role. What it does not show is the harder part, which is that every other copy of email (the raw topic, the export, the feature table) needs its own equivalent of this boundary. A masked view on one table says nothing about the rest.

Least privilege and revocation across copies

Applying least privilege to a platform means giving each identity only the operations and resource scope its work needs. Service identities need the same care as people, since a pipeline credential that can read every dataset is a way for any job to read everything. Revocation has its own timing. Removing someone from a group changes one record; whether their existing sessions, cached decisions, issued tokens, and shared file links stop working depends on how each path learns about the change. Access revocation therefore needs an intended window and a test at each boundary. And revocation cannot recall data already exported: it prevents future authorized use inside the controlled system and does nothing about a copy someone already holds.

Retention: a duration needs a starting event and an end action

Retention is the other half of the lifecycle. Data retention states why information is kept, for how long, and what happens at the end. The storage limitation principle in GDPR Article 5(1)(e) says personal data should be kept in an identifiable form for no longer than necessary for the purposes of processing, with a longer period allowed for archiving, research, or statistical purposes under safeguards. The regulation does not say how many days that is. A retention schedule is the artifact that turns the principle into numbers that a job can execute, and each row needs four things: a category defined clearly enough to decide membership, a period with the event it counts from, a stated basis, and an end action (delete, anonymize, or aggregate then delete the detail).

The retention clock is where copies break the schedule. If a retention rule counts from account closure, every copy needs access to the account-closure date. Rewriting an old record into a new file must not restart its lifetime; deriving a feature row from a customer record must not reset the clock; an export must carry the date or be tracked by something that does. This is why the metadata record above stores the clock-start field with the dataset, and why a lifecycle policy based on file creation time cannot, by itself, implement per-person retention: the file’s creation date and the person’s account-closure date are unrelated.

Retention also operates at a different layer from erasure. Retention is a rule applied to categories on a schedule. Erasure is an event triggered by a request about one person and applied before the schedule would have expired the data. The same mechanisms (delete rows, expire versions, rotate backups) serve both, but the trigger, the deadline, and the evidence differ. A platform with retention jobs and no erasure path can honor expiry and still fail a request. A platform with an erasure path and no retention schedule accumulates data nobody has a justification for.

Every layer that stores data also has its own retention parameters, and these determine how long a deleted row remains physically present. The table lists the documented defaults for the systems used as examples in this article; values were checked in early October 2026 against the vendors’ documentation and can be changed per table or account.

System	Mechanism	Documented default	Effect on a deleted row
Apache Kafka topic	cleanup.policy and delete.retention.ms	Policy delete by default; tombstones kept 1 day on compacted topics	Under compact, the latest value per key is retained until a tombstone is written and later removed; consumers reading from the start can miss the tombstone if they take longer than the retention
Delta Lake table	delta.deletedFileRetentionDuration, delta.logRetentionDuration, VACUUM	7 days for removed files; 30 days for log history; VACUUM is not run automatically	Older versions with the deleted row stay readable until files are vacuumed
Apache Iceberg table	expire_snapshots, history.expire.max-snapshot-age-ms, orphan file removal	5 days maximum snapshot age when expiring; orphan file default interval 3 days	Data files referenced by a retained snapshot are not removed
Snowflake	Time Travel, then Fail-safe	Time Travel 1 day by default, up to 90 days on Enterprise Edition and higher for permanent objects; Fail-safe 7 days, not configurable	Deleted data is recoverable by the customer during Time Travel and by the vendor during Fail-safe
BigQuery	Time travel, then fail-safe	Time travel window 7 days by default, configurable from 2 to 7 days; fail-safe 7 additional days, not configurable	Data remains available for recovery for up to the sum of the two windows
Amazon S3 with versioning	Delete markers and object versions	Versioning is off by default; when on, a plain delete adds a delete marker	Earlier versions remain until they are deleted

Read the last column as the answer to a specific question: after the row is deleted logically, how long can it be read by someone with the right privileges? Under these defaults the answer ranges from effectively zero, for an in-place delete in a row store, to more than ninety days in a warehouse configured for the maximum time-travel window plus the fail-safe period. Those numbers are the reason a deletion request does not complete when the DELETE statement does.

Derived copies: finding what the source key reached

Erasure starts with a search. Given a subject identifier, which datasets hold this person’s data? The lineage graph is the main tool: from the source table, follow edges to every downstream dataset and keep those that carry the subject key or a value derived from it. The code below is a deliberately small model of that walk over the illustrative platform. It was executed locally with Python 3.12 using only the standard library, and the graph is invented for this article.

# Illustrative platform: copies of customer data and how a subject's identifier reaches them.
# edges: (upstream, downstream, how the subject_id appears downstream)
edges = [
 ("app_db.customers",   "raw.customers_cdc",      "key"),
 ("raw.customers_cdc",  "silver.customers",       "key"),
 ("silver.customers",   "gold.revenue_by_month",  "aggregated"),
 ("silver.customers",   "ml.features_v3",         "key"),
 ("silver.customers",   "search.customer_index",  "key"),
 ("silver.customers",   "export.partner_files",   "key"),
 ("app_db.customers",   "bi.extract_manual",      "key"),     # not in lineage tool
]
catalog_known = set(edges) - {("app_db.customers","bi.extract_manual","key")}   # tool missed the manual extract

def reach(edges, start="app_db.customers"):
    seen, order, stack = {start}, [], [start]
    while stack:
        n = stack.pop()
        for u, d, how in edges:
            if u == n and d not in seen and how != "aggregated":   # aggregates need a separate decision
                seen.add(d); order.append(d); stack.append(d)
    return order
true_set  = set(reach(edges))
plan_set  = set(reach(list(catalog_known)))
print("copies that hold the key (ground truth):", len(true_set))
print("copies found by recorded lineage        :", len(plan_set))
print("missed by lineage                       :", sorted(true_set - plan_set))
print("aggregate nodes needing a decision      :", sorted(d for u,d,h in edges if h=="aggregated"))

copies that hold the key (ground truth): 6
copies found by recorded lineage        : 5
missed by lineage                       : ['bi.extract_manual']
aggregate nodes needing a decision      : ['gold.revenue_by_month']

Reading the output: the walk found five copies from the recorded edges. The sixth, a spreadsheet extract that an analyst built by hand from the source database, has no edge in the lineage tool, so the plan misses it while the truth includes it. This is the property of lineage that matters for erasure. A recorded graph is a lower bound on where data went, never a proof of where it did not go. The same output also shows the aggregate node. The walk deliberately does not follow it, because whether the monthly revenue table still identifies anyone is an assessment of group sizes and linkability, not something a graph traversal can decide.

Four practical rules follow from that. First, treat a missing edge as a normal condition. Compensate with a periodic scan that looks for the subject key or known identifier columns in datasets outside the graph, and with a rule that new datasets cannot be registered without a declared source. Second, hash or token substitution does not remove the subject from the copy: a deterministic hash of an email can be reproduced by anyone with a list of candidate addresses, so a table keyed by hashed email is still personal data to be found and handled. The pseudonymization and anonymization entry explains why. Third, derived representations such as embeddings or model parameters need a stated policy. Deleting the training rows does not demonstrate that their influence has left a trained model; that requires a separate assessment, which the data deletion entry also states. Fourth, for aggregates, record the decision with who made it, on what basis, and what would invalidate it.

Entity resolution matters here in a specific way. If the same person appears under three customer identifiers because merges were missed, a request resolved against one identifier erases a third of their data. Conversely, a wrong merge ties two people’s records together, so an erasure for one can delete the other’s. The practical response is to run the erasure against the resolved cluster of identifiers with its decision history, and to log which identifiers were included.

Deletion propagation: making each copy forget

Propagation turns an erasure request into a set of actions, one per copy, each with a mechanism, a deadline, and a verification. A useful way to think about it is that a deletion request is a durable object with a lifecycle of its own: received, scoped (which identifiers, which datasets), executing (per-destination status), waiting (for retention windows to pass), verified, and closed. GDPR Article 12(3) requires information on action taken on a request without undue delay and within one month of receipt, with a possible extension of two further months where necessary given complexity and number of requests, and the controller must inform the person of the extension and the reasons within the first month. Article 19 adds that the controller communicates the erasure to each recipient the data was disclosed to, unless that proves impossible or involves disproportionate effort. The ICO’s guidance on the right to erasure describes the same notification duty. The UK guidance page carries a notice that it is being revised following recent legislation, so check the current version.

An engineering consequence of the deadline is that the platform has to be able to say, at any point, which destinations have confirmed and which are waiting on a retention window. A single success flag is not enough, because several copies finish at different times and some by design finish days later.

Operational stores and the replication stream

In an operational database the action is a DELETE or an anonymizing UPDATE, and it completes at commit. The harder question is what the replication path does. A change-data-capture pipeline emits a delete event, and the downstream sink must apply it. On a Kafka topic with the compact cleanup policy, a deletion is represented by a record with the key and a null value, called a tombstone. Compaction retains the latest value per key, and the tombstone is itself removed after delete.retention.ms, which defaults to one day. Two consequences matter. A topic with the delete policy keeps the old values until the time or size limit discards segments, regardless of tombstones, so the original message persists for the topic’s whole retention. And a consumer that rebuilds state from offset zero may miss the tombstone if it finishes reading after the tombstone has been collected. The delete tombstone entry covers the sink side: a version-guarded destination should remember the source version of the deletion, so that an older replayed update with a lower version cannot resurrect the row.

Replay is the general hazard. Any process that re-reads history, such as a backfill, a replay of a landing zone, or a restore, can reintroduce a deleted row into a store that had removed it. The defense is to apply erasure at the point where data enters each store, not only once downstream. A suppression list of erased subject keys, consulted by ingestion and backfill jobs, is a common pattern. It is itself data about a person and must be minimized: store the key or a keyed hash, not the rest of the record, and give the list its own retention and access rules. Whether and how long such a list may be kept is a question for whoever owns the legal obligation.

Lakehouse tables: logical deletion first, physical removal later

Tables in an open table format are stored as immutable data files plus metadata that says which files make up each version. Deleting a row therefore does not edit a file. Apache Iceberg’s documented default for delete commands is copy-on-write: the files that contain the row are rewritten without it, and a new snapshot points at the new files. Merge-on-read is the alternative: the table writes a small delete record next to the data file and readers apply it. Iceberg stores such records as position delete files (format version 2) or as binary deletion vectors (version 3). Delta Lake also supports deletion vectors, which mark rows as removed without rewriting the Parquet file; its documentation says the changes are applied physically when files are rewritten by OPTIMIZE or by REORG TABLE ... APPLY (PURGE), and that modified data may still exist in old files until VACUUM removes them.

In both modes the logical delete is fast and the physical removal depends on later maintenance. For an erasure obligation, a successful DELETE only means that current queries stop returning the row. The row still exists in older files that older versions refer to, and in merge-on-read tables it also exists in the current files until they are rewritten. Completion therefore requires the rewrite (so no live file contains the row), the expiry of snapshots that reference the old files, and the physical deletion of unreferenced files. Iceberg’s expire_snapshots removes snapshots according to age settings, and its documentation states that it never removes files still required by a non-expired snapshot. Delta’s documentation adds a detail worth planning around: when REORG TABLE ... APPLY (PURGE) runs, it creates a new table version, and that timestamp, not the original delete time, is the one to consider for the VACUUM retention threshold.

The following script shows the sequence on a real Delta table using the deltalake Python package (the delta-rs implementation, version 1.6.6) on local files. It is a demonstration of the mechanism, not a statement about every engine. Its behavior differs from Spark-based Delta in details, and disabling the retention check, as the script does to show the end state, is something production systems do only with care. The data is invented (three rows with placeholder addresses).

import os, shutil, glob
import pyarrow as pa, pyarrow.parquet as pq
from deltalake import DeltaTable, write_deltalake

root = "work/payments_customers"
shutil.rmtree("work", ignore_errors=True); os.makedirs("work")
rows = pa.table({"customer_id": [1, 2, 3],
                 "email": ["a@example.test", "b@example.test", "c@example.test"],
                 "plan": ["basic", "pro", "basic"]})
write_deltalake(root, rows)                      # version 0
# a derived copy made before the erasure request (e.g., an export for a partner)
pq.write_table(rows, "work/partner_export.parquet")

dt = DeltaTable(root)
dt.delete("customer_id = 2")                      # version 1
dt = DeltaTable(root)

def data_files(): return sorted(os.path.basename(p) for p in glob.glob(root + "/*.parquet"))
def emails(t): return sorted(t.to_pyarrow_table()["email"].to_pylist())

print("current version:", dt.version())
print("current read     :", emails(dt))
print("time travel v0   :", emails(DeltaTable(root, version=0)))
print("files on disk    :", len(data_files()))
print("partner export   :", sorted(pq.read_table("work/partner_export.parquet")["email"].to_pylist()))

print("-- vacuum dry run, default retention (7 days):")
print("   would delete:", len(dt.vacuum(dry_run=True)))
print("-- vacuum retention 0h, retention check disabled:")
gone = dt.vacuum(retention_hours=0, dry_run=False, enforce_retention_duration=False)
print("   deleted files:", len(gone))
print("files on disk    :", len(data_files()))
try:
    print("time travel v0   :", emails(DeltaTable(root, version=0)))
except Exception as e:
    print("time travel v0 failed:", type(e).__name__)
print("current read     :", emails(DeltaTable(root)))
print("partner export   :", sorted(pq.read_table("work/partner_export.parquet")["email"].to_pylist()))
import deltalake; print("deltalake", deltalake.__version__)

current version: 1
current read     : ['a@example.test', 'c@example.test']
time travel v0   : ['a@example.test', 'b@example.test', 'c@example.test']
files on disk    : 2
partner export   : ['a@example.test', 'b@example.test', 'c@example.test']
-- vacuum dry run, default retention (7 days):
   would delete: 0
-- vacuum retention 0h, retention check disabled:
   deleted files: 1
files on disk    : 1
time travel v0 failed: FileNotFoundError
current read     : ['a@example.test', 'c@example.test']
partner export   : ['a@example.test', 'b@example.test', 'c@example.test']
deltalake 1.6.6

What the output shows, line by line. After the delete, the current read returns two rows, but reading version 0 still returns all three, so a reader with time-travel access can still see the erased address. Two data files exist on disk: the original and the rewritten one. A dry-run VACUUM with the default seven-day retention would delete nothing, since the old file is too recent, which is the situation a team is in during the first week after an erasure. After a VACUUM with zero retention, one file remains, version 0 can no longer be read because its data file is gone, and the current table is unchanged. The last line is the other half of the lesson: the export written before the request still contains all three addresses, because it was a copy outside the table. Table maintenance cannot reach it.

Warehouses, object stores, and search

Managed warehouses hide the file mechanics but keep the same structure. In Snowflake a deleted row stays recoverable through Time Travel for the configured period and then through Fail-safe for 7 days, which cannot be changed and which is a recovery path operated by the vendor and not by customers. BigQuery documents a time travel window of 7 days by default, configurable from 2 to 7, followed by a non-configurable 7-day fail-safe. The design lever the platform owns is the Time Travel window, and it is a trade-off between the ability to undo mistakes and the speed of physical erasure. The time travel entry notes that retention is not free, since prior versions occupy storage. For tables that hold personal data, set the window deliberately instead of accepting a setting chosen for convenience, and record in the erasure procedure the date on which the last recoverable copy will expire.

For object storage with versioning, a plain delete adds a delete marker, and earlier versions remain until they are deleted. Permanent removal on such a bucket has to address the versions and not only the current object, and if Object Lock has been applied, locked versions cannot be removed during their retention period or legal hold. That is an engineering reason not to put personal data in locked buckets without having decided how a conflicting erasure request will be handled. For search indexes and caches, the action is a delete by document key with a retry loop. As the data deletion entry points out, if the index deletion fails, restrict serving of the document until it is repaired instead of announcing completion.

Soft deletion, where a row is marked deleted and retained for a recovery window, is a legitimate design for protecting against accidental deletion. It also means data exists in a retrievable state for a stated period, and the soft deletion entry notes that the point is two promises: the data becomes unusable immediately, and it is destroyed after a finite delay. A purge job that never runs turns a recovery feature into silent retention, so the purge deserves monitoring like any other job.

Crypto-erase: destroying a key instead of rewriting data

For stores where individual rows cannot be cheaply removed, such as large immutable files and backups, an alternative is to encrypt each subject’s data under a key of their own and to erase by destroying the key. NIST’s media sanitization guidance defines cryptographic erase as a purge technique in which key sanitization is applied to the keys that provide confidentiality for the encrypted data, making recovery of the decrypted data infeasible. The mechanics are those of envelope encryption with the data key scoped to the person. The script below uses the Python cryptography package (version 50.0.2) with Fernet to show the structure on three invented subjects. Fernet is used for brevity; a production design would use a key management service.

from cryptography.fernet import Fernet, InvalidToken
import cryptography

keys = {}                                   # stands in for a key service: subject_id -> data key
def enc(subject, text):
    k = keys.setdefault(subject, Fernet.generate_key())
    return Fernet(k).encrypt(text.encode())
def dec(subject, blob):
    if subject not in keys: raise KeyError("no key for subject")
    return Fernet(keys[subject]).decrypt(blob).decode()

live   = {s: enc(s, f"{s}@example.test") for s in ("u1", "u2", "u3")}
backup = dict(live)                         # a backup copies ciphertext only
print("before:", dec("u2", backup["u2"]))
del keys["u2"]                              # erasure = destroy the subject's key
for name, store in (("live", live), ("backup", backup)):
    try: dec("u2", store["u2"])
    except KeyError as e: print(name, "u2 ->", e)
print("u1 still readable:", dec("u1", backup["u1"]))
print("cryptography", cryptography.__version__)

before: u2@example.test
live u2 -> 'no key for subject'
backup u2 -> 'no key for subject'
u1 still readable: u1@example.test
cryptography 50.0.2

The output shows the property that makes the approach attractive: after the key for u2 is deleted, the live store and a backup that holds only ciphertext both fail to produce the value, while the other subjects remain readable. The conditions under which this is true are strict. Every copy of the key must be destroyed, including wrapped keys held in backups and caches. The data must only exist as ciphertext in the places claimed, so a plaintext extract made earlier by an authorized job is not covered. And the cost is operational: one key per subject means millions of keys to store, authorize, and audit, so the design is usually applied to the stores where it is needed, not everywhere. The envelope encryption entry notes that deleting one current wrapper cannot make retained copies unreadable if other copies of the wrapped key persist.

Putting these mechanisms together, the table below turns the illustrative platform into an erasure plan. The “earliest physical removal” column uses the documented defaults from the retention table plus an assumed daily maintenance job and is arithmetic on stated assumptions, not a measurement of any system.

Copy	Action on request	Earliest physical removal (assumptions)	Verification
Operational table	Delete or anonymize by subject key	At commit	Query by key returns no rows
Raw CDC topic (delete policy)	Cannot edit; rely on topic retention or republish a filtered topic	When retention discards the segment (topic-specific)	Check retention setting and oldest offset time
Silver table (Delta, default retention)	DELETE, then purge and VACUUM	About 7 days after the purge version if VACUUM runs daily	Read oldest retained version; list data files
Warehouse table (30-day Time Travel)	DELETE	30 days of Time Travel plus 7 days Fail-safe, 37 days	Time Travel query at the oldest retained time
Feature table	Delete rows by key, retrain on the next scheduled run	Next training run after deletion	Training dataset snapshot excludes the key
Search index	Delete by document key with retries	Immediate on success	Query and known-key probe
Partner export	Notify recipient; contractual deletion	Outside platform control	Recipient confirmation recorded
Manual extract	Found by scan; owner deletes	Depends on owner	Scan finds no key
Backups	Beyond use now; expire on schedule; reapply on restore	Up to backup retention, for example 35 days	Restore drill applies the erasure list

Backup exceptions: bounded, documented, and reapplied on restore

Backups are the copies that conflict most directly with erasure, because their value is that they do not change. Regulators address this explicitly, and the answers are narrower than “backups are exempt”. The ICO states that where an erasure request is valid and no exemption applies, steps must be taken to erase from backup systems as well as live ones, and that the steps depend on the retention schedule for backups and the technical mechanisms available. It describes the key issue as putting the data beyond use even if it cannot be immediately overwritten: the backup is held until it is replaced on an established schedule and is not used for any other purpose. It also says organizations must be clear with individuals about what happens to their data in backups. California’s regulations on deletion requests take a related position: a business that stores personal information on archived or backup systems may delay compliance with respect to that data until the backup is restored to an active system or next accessed or used for a sale, disclosure, or commercial purpose.

Both positions share three engineering conditions, which turn a vague exception into a design.

• Bounded. Backups expire on a fixed schedule, so the longest period an erased record can survive is the backup retention plus the time to expire the last copy. With 35-day retention (an assumed figure) a record erased on day 10 can persist in a backup taken on day 3 until day 38. Retention of backups is therefore a privacy parameter, not only a recovery parameter.
• Beyond use. The backup is not queried, mounted for analytics, or used to seed a development environment. Access to backups needs its own authorization, and “restore to test” workflows are the usual place where beyond-use quietly stops being true.
• Reapplied on restore. The platform keeps a record of erasures executed after each backup was taken, and a restore applies that record before the restored data becomes available to anyone. Without it, a restore silently reintroduces erased records.

The third condition is the one that needs code. The fragment below models it with invented data. It was executed locally with the standard library.

from datetime import date, timedelta
backup_retention_days = 35
erasure_log = {"u2": date(2024, 11, 10)}            # subject -> day the erasure was executed in live systems
backups = {date(2024, 11, 3): {"u1","u2","u3"},      # backup taken before the erasure
           date(2024, 11, 12): {"u1","u3"}}          # backup taken after
def restore(day):
    rows = set(backups[day])
    print(f"restore {day}: raw rows = {sorted(rows)}")
    # reapply erasures that happened after the backup was taken
    for subject, erased_on in erasure_log.items():
        if erased_on > day and subject in rows:
            rows.discard(subject); print(f"  reapplied erasure for {subject}")
    return rows
print(sorted(restore(date(2024, 11, 3))))
print(sorted(restore(date(2024, 11, 12))))
print("oldest backup that can still hold u2 expires on:",
      date(2024,11,3) + timedelta(days=backup_retention_days))

restore 2024-11-03: raw rows = ['u1', 'u2', 'u3']
  reapplied erasure for u2
['u1', 'u3']
restore 2024-11-12: raw rows = ['u1', 'u3']
['u1', 'u3']
oldest backup that can still hold u2 expires on: 2024-12-08

The output shows the difference between the two backups. The earlier backup was taken before the erasure, so its raw rows include u2, and the restore procedure removes the subject before returning the data. The later backup never contained the subject, so nothing needs to be reapplied. The last line computes the date on which the oldest backup that could hold the subject expires, which is the date the erasure plan records as the end of the backup exception. Two caveats: the erasure list is itself personal data to be minimized (a key or keyed hash, not the record), and it must itself survive the incident that causes the restore. A list stored only in the system being restored is useless.

Other exceptions and conflicts

Erasure is not unconditional. GDPR Article 17(3) lists situations in which the right does not apply to the extent processing is necessary, including compliance with a legal obligation, the establishment, exercise, or defense of legal claims, and archiving, research, or statistical purposes where erasure would seriously impair those objectives. In practice the platform meets these as exceptions on specific datasets for stated reasons, and as a legal hold on specific content for a specific matter. A hold preserves the content and versions in scope, so it conflicts with deletion by design. The decision of which prevails for a given record is a legal determination, made per case. The platform’s job is to represent the outcome: the request recorded, the decision and its reasoning recorded, the affected data identified, the disposition reassessed when the hold lifts, and deletion performed once no other hold or retention requirement remains. Releasing one hold removes only that hold’s restriction, so a record covered by two holds stays preserved when one is released.

Exceptions need the same discipline as any other control exception. Each has a scope (which datasets or records), a reason, an owner, and a review date, and held data remains governed data. A common failure is an extract made “for the lawyers” and left in a location nothing manages; it becomes the least protected copy of the most sensitive data.

Audit evidence: showing what was done without copying the data

Because erasure is claimed rather than observed, the platform needs evidence. An audit log that supports erasure records, for each request, the request identifier, the identifiers resolved, the destinations in scope, the action taken and its outcome per destination, the time, and who or what executed and verified it. It should record the action and outcome without copying the deleted payload. A log line that says “deleted email b@example.test” re-creates the data it reports on. Prefer identifiers, counts, and status codes, restrict access to the log, and apply retention to it as well.

Verification is a separate step from execution. A query returning no rows proves what that interface returned. For stronger evidence, combine several checks: the destination’s own delete confirmation, a negative query by subject key, a scan for alternative identifiers such as the email address, and where relevant an inspection of the physical state (files containing the row, snapshots still referencing them). A known-subject probe is useful for continuous checking: periodically create a synthetic subject, run it through an erasure, and confirm that every registered destination forgets it. The probe finds destinations whose delete path has silently stopped working. Mismatches are expected, and the process needs a stated response: repair the failing destination, restrict serving of the affected data in the meantime, and extend the request status instead of closing it.

Because ABAC decisions depend on attributes that change, access evidence has its own requirement: retain the policy version and the attribute values relevant to a decision, so that a past access can be explained. Without it, the audit log can show that access happened, and nothing about why it was allowed. For the retention side, continuous control monitoring is the idea of measuring the control’s operation on a schedule instead of sampling it at audit time.

Measure	What it tells you	Limit
Share of datasets with an owner and classification	Coverage of the catalog	Says nothing about label accuracy; sample and test
Erasure requests closed within the response period	Operational performance	A closed request is only as good as the scope and verification
Destinations by delete mechanism (physical, deferred, key destroyed, notify only)	Where completion depends on a clock or a third party	Needs the inventory of destinations to be complete
Lineage gaps found by identifier scans	How incomplete the graph is	Finds what the scan can see; free text and unrecognized identifiers escape it
Age of the oldest backup that can hold an erased subject	Length of the backup exception	Depends on the backup catalog being accurate
Probe subject deleted in every registered destination	Whether each delete path still works	Synthetic subject may not exercise every code path

Failure modes and limits

• Complete labels, incomplete enforcement. Labels in a catalog protect nothing unless an enforcement point reads them. Test access, retention, and deletion on new derivatives and not only on the labeled source.
• The copy outside the graph. Manual extracts, ad hoc notebooks, and exports have no edges. Identifier scans and a registration rule reduce this and do not eliminate it.
• Resurrection. Replays, backfills, and restores can reintroduce erased rows. Apply suppression at ingestion and on restore.
• Completion declared too early. A successful DELETE statement is not completion on immutable-file formats or in warehouses with time travel. Track physical removal separately.
• Retention that nothing executes. A schedule that is documented and not run accumulates data and evidences a rule being ignored. Retention jobs should emit counts, and the counts should be read.
• Evidence that re-creates the data. Logs, probes, and suppression lists contain identifiers by nature. Minimize them and give them retention rules.
• Unresolved identity. If entity resolution has missed a merge, the erasure reaches only part of a person’s data; if it merged wrongly, it can reach another person’s. Run erasure against the resolved cluster and log which identifiers it used.

None of the mechanisms here is a guarantee on its own. Together they give a platform a defensible answer to three questions: where is this person’s data, what happens to each copy, and what proof exists. The honest statement of the limits (outbound copies, backup windows, learned parameters, unresolved identities) is part of the answer, because the person and the regulator are owed an accurate account and not an announcement that everything is gone.

The next article, on data SLIs, SLOs, and incident response, moves from controlling what the platform stores to measuring and defending the reliability of what it delivers: Data SLIs, SLOs, and Incident Response.

References

• GDPR Article 17, Right to erasure (text as reproduced at gdpr-info.eu)
• GDPR Article 12, Transparent information and modalities for exercising rights (gdpr-info.eu)
• GDPR Article 19, Notification obligation regarding erasure (gdpr-info.eu)
• GDPR Article 5, Principles relating to processing of personal data (gdpr-info.eu)
• GDPR Article 4, Definitions (gdpr-info.eu)
• ICO, Right to erasure
• California Code of Regulations, Title 11, section 7022, Requests to Delete (Cornell LII)
• NIST CSRC Glossary, cryptographic erase (source: NIST SP 800-88 Rev. 2)
• Delta Lake documentation, Table utility commands (VACUUM, RESTORE)
• Delta Lake documentation, Table batch reads and writes (time travel and data retention)
• Delta Lake documentation, What are deletion vectors?
• Apache Iceberg documentation, Maintenance
• Apache Iceberg documentation, Configuration
• Apache Iceberg Table Spec
• Apache Kafka documentation, Topic configs
• Snowflake documentation, Understanding and using Time Travel
• Snowflake documentation, Understanding and viewing Fail-safe
• Google Cloud, BigQuery data retention with time travel and fail-safe
• Amazon S3 User Guide, Retaining multiple versions of objects with S3 Versioning
• Amazon S3 User Guide, Locking objects with Object Lock