Storage Tiering, Lifecycle, and Archival

In plain terms

Retention asks whether data should remain; tiering asks how to store the data that remains. An archive can reduce storage charges while increasing retrieval cost and recovery time. Neither age nor a low price establishes that data should be kept, deleted, or archived. Start with dataset owners, permitted retention, measured access, and the recovery deadline.

The previous article established immutable runs, publication references, and safe cleanup. This article adds tier residence, lifecycle filters, minimum-duration charges, restore planning, and retention controls. The Python lab models synthetic access and prices. It performs local checks on sample bytes, not a cloud restore or a compliance assessment.

Tier names do not imply identical behavior

Compare storage price, retrieval and request charges, minimum billable size and duration, availability and redundancy, and time to readable data. Hot and online infrequent-access tiers support direct reads; offline archives require a restore or rehydration workflow. A single-zone tier changes the resilience boundary as well as price, so it is not merely a cheaper equivalent of a multi-zone tier.

GCS Archive remains online and has a 365-day minimum storage duration. S3 Glacier Flexible Retrieval and Deep Archive need restoration; Deep Archive bulk retrieval typically completes within 48 hours, rather than the one-day ceiling in the original draft. Azure Archive requires rehydration and has a 180-day minimum. These are distinct service behaviors, not the lab’s generic five- and twelve-hour assumptions. Consult GCS storage classes, S3 archived objects, S3 restore options, and Azure access tiers.

Automatic placement such as Intelligent-Tiering or Autoclass has its own eligible sizes, monitoring charges, and retrieval/transition rules. Access-based lifecycle conditions also depend on the product and configuration. Do not transfer one service’s thresholds or billing exceptions to another. Prices must be checked for the selected region, redundancy, tier, and account before deployment.

Access evidence and the full cost comparison

Measure requests, bytes read, range-read patterns, repeat access, and latency requirements per dataset. A yearly audit and a daily dashboard can read the same old table for different reasons. A request count alone cannot estimate retrieval GB, and a short log window can miss seasonal work. Record the observation period and known future backfills.

The lab combines 20,000 generated read records with one full-history audit sweep. Its 92% old-byte share and 14% read share are properties of that sample. They motivate comparison but do not establish the cheapest tier. Keeping the small curated table online may be worthwhile even when raw replay sources can tolerate a wait.

For a fixed window, compare annual storage savings with the extra cost of reading from archive instead of the alternative tier. Both sides may charge for retrieval. Add request fees, restored-copy storage, egress, compute, minimum-duration penalties, transition charges, and the value of meeting the recovery deadline. The lab’s simplified break-even is annual storage savings divided by the difference in per-read retrieval fees. It excludes those other costs, assumes repeated full-window reads, and holds prices and tiers fixed for the year.

Lifecycle clocks, filters, and safe expiration

Distinguish object creation or modification time, time entered into a tier, time a version became noncurrent, and time a snapshot stopped being referenced. They answer different questions. A 1,000-day-old object moved to cool today has zero days of cool residence. In a simplified 30-day minimum model, deleting it after 15 days adds 15 days of cool storage charges; it does not mean the first 30 hot days were also charged at the cool rate.

Rules combine supported filters and time conditions. Evaluation and physical execution are asynchronous, not a guaranteed deletion at midnight. Prefixes are literal strings unless the API explicitly provides pattern syntax. Tags are metadata; they protect nothing unless a rule or enforcement mechanism uses them. S3 lifecycle rules includes object-size filters and a default restriction on transitions below 128 KB. Azure lifecycle management describes access tracking and policy execution. Exact supported transitions and minimum-duration billing require the chosen provider’s rules.

A direct migration of an existing old object is not necessarily billed as every historical tier hop. The lab chooses the deepest eligible target once, records entry today, and does nothing on a repeated evaluation. It does not implement asynchronous provider execution or transition eligibility checks. Its output is a plan and storage run rate, not a realized invoice.

Noncurrent version retention uses the date the version became noncurrent, not the object’s creation age. Preserve referenced versions and recovery requirements before expiring them; thirty days is a candidate, not always sufficient. Ordinary current-object listings can omit older versions, but version-aware inventory can expose them. Protect the manifest and every data object reachable from retained snapshots or active readers, plus active writers. Keeping a pointer while deleting its target still breaks the table. Prefix scoping alone is not a complete garbage collector.

Restore planning and a meaningful drill

Before archiving, define the recovery time objective and acceptable recovery point. Keep an online inventory of keys, versions, tier, sizes, checksums, schema and decoder versions, and required encryption keys. The restore runbook requests a supported retrieval mode, tracks completion and failures, reads before any temporary copy expires, verifies content, and runs a real consumer. Time the entire process, including queueing, download, decryption, parsing, and rebuilding downstream data.

For S3 Flexible Retrieval and Deep Archive, restoration exposes a temporary readable copy for a chosen duration; the archive remains. Azure rehydration and GCS direct reads follow different workflows. A modeled wait is not a service-level guarantee. Budget restored-copy storage and repeat requests, and avoid a lifecycle rule immediately re-archiving newly recovered data.

The local drill below hashes stored JSON bytes, corrupts one returned payload, and checks size, full SHA-256, and consumer parsing/schema. It tests the validation logic. It neither waits for a cloud restore nor establishes the recoverability of every archived object. A real drill samples across sources, ages, sizes, versions, encryption keys, and recovery workflows; findings remain attached to the exact object version until resolved. Archiving the only copy is not by itself an independent backup.

Retention, holds, and deletion

Document why a dataset is retained, who can approve changes, and when retention is reviewed. Service-enforced retention or legal holds are stronger than a descriptive tag. For example, S3 Object Lock considerations explains interactions with lifecycle and object versions: protected versions are not simply erased by expiration, while delete markers can affect ordinary visibility. Test the selected enforcement mode and authorized release process. A business hold represented only in a separate registry must be integrated into deletion planning.

An erasure request concerning rows inside shared files requires data lineage across raw, curated, versions, replicas, indexes, and backups. Rewriting affected files can remove rows from current datasets, but old versions and snapshots still need disposition. Tombstones suppress reads only if every relevant access path honors them; they are not physical erasure. Per-person encryption can support cryptographic erasure only if keys, wrapped copies, caches, and plaintext derivatives are actually controlled. It is not generally one API call.

A three-year expiry or a sentence in a privacy notice does not establish an acceptable response to a deletion request. Determine applicable deadlines, lawful retention exceptions, and treatment of backups with the responsible privacy/legal owners. The ICO erasure guidance distinguishes live systems, backups, and exceptions; it is jurisdiction-specific guidance, not a universal tombstone policy. Record and verify the approved outcome. The lab only estimates a synthetic rewrite footprint and filters a supplied hold set.

Operating a reviewed policy

Keep policy configuration and its rationale in version control. Before enabling a destructive rule, compute candidate objects and versions from an inventory, inspect samples, apply reference and hold exclusions, estimate charges, and test on a disposable dataset. A local dry-run simulator is not evidence that the cloud service will choose the same objects. Roll out narrowly with owner review, monitor actual actions, and retain the policy version and candidate report.

Track bytes and versions per tier, transition and expiration counts, restore success and elapsed time, costs, and missed freshness or recovery objectives. Investigate unexpected changes rather than assuming every spike is a bad prefix. Disabling a rule may stop future scheduling but cannot undo completed deletion or guarantee that already queued work is cancelled. Review access patterns when consumers or legal requirements change, and keep restoration tests separate from destructive policy tests.

Lab

Run both setup blocks before an exercise in Python 3.12. The model creates 1,095 daily partitions per table, uses decimal GB, and assigns synthetic read records and separately modeled noncurrent dates. Generic tier prices and delays are invented. monthly_cost is current-byte storage run rate only; noncurrent storage and other charges are separate. No provider rules are deployed, and no real data is deleted or restored.

import hashlib, random
from collections import Counter, defaultdict
from datetime import date, timedelta

rng = random.Random(0)
today = date(2026, 3, 1)
GB = 1_000_000_000
TIERS = {
    "hot":     {"storage": 0.023, "retrieval": 0.0,  "min_days": 0,   "restore_hours": 0,  "transition_per_1000": 0.0},
    "cool":    {"storage": 0.0125, "retrieval": 0.01, "min_days": 30,  "restore_hours": 0,  "transition_per_1000": 0.01},
    "archive": {"storage": 0.004, "retrieval": 0.02, "min_days": 90,  "restore_hours": 5,  "transition_per_1000": 0.03},
    "deep":    {"storage": 0.001, "retrieval": 0.02, "min_days": 180, "restore_hours": 12, "transition_per_1000": 0.05},
}

partitions = []
for table, objects_per_day, gb_per_day in (("raw.events", 24, 2.0), ("raw.orders", 4, 0.3), ("curated.daily_revenue", 1, 0.01)):
    for age in range(1, 3 * 365 + 1):
        day = today - timedelta(days=age)
        partitions.append({"table": table, "day": day, "age": age, "objects": objects_per_day, "bytes": int(gb_per_day * GB),
                           "tier": "hot", "tier_since": day, "noncurrent_since": today - timedelta(days=min(age, age % 60 + 1)), "noncurrent_bytes": int(gb_per_day * GB * 0.1) if age % 7 == 0 else 0})

reads = []
for _ in range(20000):
    roll = rng.random()
    if roll < 0.80:
        age = rng.randint(1, 30)
    elif roll < 0.95:
        age = rng.randint(31, 120)
    else:
        age = rng.randint(121, 3 * 365)
    reads.append({"table": rng.choice(["raw.events", "raw.events", "raw.orders", "curated.daily_revenue"]), "age": age})
for age in range(1, 3 * 365 + 1):
    reads.append({"table": "curated.daily_revenue", "age": age})

def monthly_cost(parts):
    """Storage only: each partition's bytes at its tier's price per GB-month."""
    return sum(p["bytes"] / GB * TIERS[p["tier"]]["storage"] for p in parts)

print(len(partitions), "partitions,", sum(p["bytes"] for p in partitions) // GB, "GB current,", sum(p["noncurrent_bytes"] for p in partitions) // GB, "GB in noncurrent versions")
print(f"all in the hot tier: {monthly_cost(partitions):,.0f} a month at the illustrative prices")
# 3285 partitions, 2529 GB current, 36 GB in noncurrent versions
# all in the hot tier: 58 a month at the illustrative prices

The lifecycle engine below applies a list of rules to the partitions and counts the transition requests it would make. Exercises 2, 3, and 6 use its result.

import copy

def apply_lifecycle(parts, rules, as_of=today, tables=None):
    """Plan one direct transition to the age-eligible target, not historical hops."""
    rank = {tier: n for n, tier in enumerate(TIERS)}
    last_day, last_rank = -1, 0
    for tier, after_days in rules:
        if tier not in rank or type(after_days) is not int or after_days <= last_day or rank[tier] <= last_rank:
            raise ValueError("rules must increase in age and tier order")
        last_day, last_rank = after_days, rank[tier]
    result, transitions = copy.deepcopy(parts), Counter()
    for p in result:
        if tables is not None and p["table"] not in tables:
            continue
        age = (as_of - p["day"]).days
        if age < 0:
            raise ValueError("future object")
        target = "hot"
        for tier, after_days in rules:
            if age >= after_days:
                target = tier
        if rank[target] > rank[p["tier"]]:
            transitions[target] += p["objects"]
            p["tier"], p["tier_since"] = target, as_of
    return result, transitions

RULES = [("cool", 30), ("archive", 180), ("deep", 730)]
tiered, transitions = apply_lifecycle(partitions, RULES)

1. Measure the synthetic access mix. Compute age-based read shares and the share of bytes older than 90 days.

def share_of_reads_older_than(days, table=None):
    pool = [r for r in reads if table is None or r["table"] == table]
    return sum(r["age"] > days for r in pool) / len(pool) if pool else None

print("share of reads touching data older than 30 / 90 / 365 days, all tables:",
      " / ".join(f"{share_of_reads_older_than(d):.1%}" for d in (30, 90, 365)))
for table in ("raw.events", "raw.orders", "curated.daily_revenue"):
    print(f"{table:22} reads older than 90 days: {share_of_reads_older_than(90, table):5.1%}, older than 365: {share_of_reads_older_than(365, table):5.1%}")
print("bytes older than 90 days:", f"{sum(p['bytes'] for p in partitions if p['age'] > 90) / sum(p['bytes'] for p in partitions):.0%}", "of the lake")
# share of reads touching data older than 30 / 90 / 365 days, all tables: 24.2% / 14.0% / 6.9%
# raw.events             reads older than 90 days:  9.6%, older than 365:  3.6%
# raw.orders             reads older than 90 days: 10.0%, older than 365:  3.4%
# curated.daily_revenue  reads older than 90 days: 24.9%, older than 365: 15.4%
# bytes older than 90 days: 92% of the lake

2. Price a direct migration plan. Choose age-eligible targets once, compare storage run rates, and repeat the evaluation.

print(f"hot everywhere: {monthly_cost(partitions):.2f} a month; tiered: {monthly_cost(tiered):.2f} a month")
print("bytes by tier after the rules:", {t: f"{sum(p['bytes'] for p in tiered if p['tier'] == t) / GB:,.0f} GB" for t in TIERS})
print("direct migration requests:", dict(transitions), "costing",
      f"{sum(n / 1000 * TIERS[t]['transition_per_1000'] for t, n in transitions.items()):.2f}")
small_files = [dict(p, objects=p["objects"] * 60) for p in partitions]
_, many = apply_lifecycle(small_files, RULES)
print("the same lake with 60 times as many objects would pay", f"{sum(n / 1000 * TIERS[t]['transition_per_1000'] for t, n in many.items()):.2f}", "to transition once")
filtered, _ = apply_lifecycle(partitions, RULES, tables={"raw.events", "raw.orders"})
print(f"keeping curated hot: {monthly_cost(filtered):.2f} monthly storage run rate")
_, repeat = apply_lifecycle(tiered, RULES)
print("repeat evaluation requests:", dict(repeat))
# hot everywhere: 58.18 a month; tiered: 11.80 a month
# bytes by tier after the rules: {'hot': '67 GB', 'cool': '346 GB', 'archive': '1,270 GB', 'deep': '845 GB'}
# direct migration requests: {'cool': 4350, 'archive': 15950, 'deep': 10614} costing 1.05
# the same lake with 60 times as many objects would pay 63.16 to transition once
# keeping curated hot: 12.00 monthly storage run rate
# repeat evaluation requests: {}

3. Compare retrieval and residence costs. Include both tiers’ retrieval charges and calculate remaining minimum-duration charges from tier entry.

window = [p for p in tiered if p["table"] == "raw.events" and 200 <= p["age"] < 290]
gb = sum(p["bytes"] for p in window) / GB
tier = window[0]["tier"]
monthly_saving = gb * (TIERS["cool"]["storage"] - TIERS[tier]["storage"])
archive_retrieval = gb * TIERS[tier]["retrieval"]
cool_retrieval = gb * TIERS["cool"]["retrieval"]
incremental_retrieval = archive_retrieval - cool_retrieval
break_even = 12 * monthly_saving / incremental_retrieval
print(f"window: {gb:.0f} GB; archive retrieval {archive_retrieval:.2f}; cool retrieval {cool_retrieval:.2f}; assumed wait {TIERS[tier]['restore_hours']} h")
print(f"monthly storage saving {monthly_saving:.2f}; extra retrieval per read {incremental_retrieval:.2f}; break-even {break_even:.1f} reads/year")

def early_exit_charge(part, as_of):
    residence = (as_of - part["tier_since"]).days
    if residence < 0:
        raise ValueError("exit before tier entry")
    tier = TIERS[part["tier"]]
    remaining = max(0, tier["min_days"] - residence)
    return part["bytes"] / GB * tier["storage"] * remaining / 30

cool_parts = [p for p in tiered if p["tier"] == "cool"]
print("newly migrated cool partitions inside minimum:", sum(early_exit_charge(p, today) > 0 for p in cool_parts))
example = dict(cool_parts[0], bytes=GB, tier_since=today - timedelta(days=15))
print(f"one GB leaving cool after 15 days: remaining-duration charge {early_exit_charge(example, today):.5f}")
# window: 180 GB; archive retrieval 3.60; cool retrieval 1.80; assumed wait 5 h
# monthly storage saving 1.53; extra retrieval per read 1.80; break-even 10.2 reads/year
# newly migrated cool partitions inside minimum: 450
# one GB leaving cool after 15 days: remaining-duration charge 0.00625

4. Versions and referenced objects. Use noncurrent dates, then compare broad expiry, prefix-only expiry, and protected candidates.

noncurrent = sum(p["noncurrent_bytes"] for p in partitions) / GB
kept = sum(p["noncurrent_bytes"] for p in partitions if (today - p["noncurrent_since"]).days < 30) / GB
print(f"noncurrent storage: {noncurrent:.3f} GB; hot run rate {noncurrent * TIERS['hot']['storage']:.2f}/month")
print(f"keep versions noncurrent for less than 30 days: {kept:.3f} GB")
lake_keys = {f"curated/daily_revenue/run_id=r{p['age']:04d}/part-0000.parquet": p["age"] for p in partitions if p["table"] == "curated.daily_revenue"}
manifest_key = "curated/daily_revenue/_manifest.json"
lake_keys[manifest_key] = 400
referenced = {"curated/daily_revenue/run_id=r0400/part-0000.parquet"}
active = {"curated/daily_revenue/run_id=r0500/part-0000.parquet"}

def expire(keys, prefix, older_than_days, protected=frozenset()):
    if type(older_than_days) is not int or older_than_days < 0:
        raise ValueError("expiry threshold must be a nonnegative integer")
    if any(type(age) is not int or age < 0 for age in keys.values()):
        raise ValueError("inventory ages must be nonnegative integers")
    return {k: age for k, age in keys.items() if k in protected or not (k.startswith(prefix) and age > older_than_days)}

broad = expire(lake_keys, "curated/daily_revenue/", 365)
scoped = expire(lake_keys, "curated/daily_revenue/run_id=", 365)
safe = expire(lake_keys, "curated/daily_revenue/run_id=", 365, referenced | active)
print("broad deleted:", len(lake_keys) - len(broad), "manifest missing:", manifest_key not in broad)
print("scoped keeps manifest:", manifest_key in scoped, "but loses referenced part:", not referenced.issubset(scoped))
print("protected selection deletes:", len(lake_keys) - len(safe), "referenced and active retained:", (referenced | active).issubset(safe))
# noncurrent storage: 36.036 GB; hot run rate 0.83/month
# keep versions noncurrent for less than 30 days: 17.556 GB
# broad deleted: 731 manifest missing: True
# scoped keeps manifest: True but loses referenced part: True
# protected selection deletes: 728 referenced and active retained: True

5. Holds and rewrite scope. Filter a supplied hold set and estimate a hypothetical user’s affected partitions.

held = {p["day"] for p in partitions if p["table"] == "raw.orders" and date(2023, 9, 1) <= p["day"] <= date(2023, 11, 30)}
expired_by_rule = [p for p in partitions if p["age"] > 365 * 2 and p["table"] == "raw.orders"]
print(len(expired_by_rule), "raw.orders partitions are past a 2-year expiry;", sum(p["day"] in held for p in expired_by_rule), "of them are under a legal hold and must be skipped by the rule")

user_partitions = [p for p in partitions if p["table"] == "raw.events" and hashlib.md5(f"U4471:{p['day']}".encode()).digest()[0] < 40]
print("a deletion request for one user touches", len(user_partitions), "immutable raw.events partitions holding", f"{sum(p['bytes'] for p in user_partitions) / GB:,.0f} GB;",
      "rewriting them all is", f"{sum(p['objects'] for p in user_partitions):,}", "objects to read and write")
print("expiry-only wait for newest affected partition:", 3 * 365 - min(p["age"] for p in user_partitions), "days; this does not establish an acceptable deletion deadline")
allowed_expirations = [p for p in expired_by_rule if p["day"] not in held]
print("hold-aware candidate expirations:", len(allowed_expirations))
# 365 raw.orders partitions are past a 2-year expiry; 91 of them are under a legal hold and must be skipped by the rule
# a deletion request for one user touches 183 immutable raw.events partitions holding 366 GB; rewriting them all is 4,392 objects to read and write
# expiry-only wait for newest affected partition: 1083 days; this does not establish an acceptable deletion deadline
# hold-aware candidate expirations: 274

6. Validate restored sample bytes. Check size, full checksum, and consumer parsing for five local sample payloads.

import json
archive_bytes = {}
for p in tiered:
    if p["table"] == "raw.events" and p["tier"] in ("archive", "deep"):
        key = f"raw/events/ingest_date={p['day']}/part-0000.json"
        archive_bytes[key] = json.dumps({"schema_version": 1, "day": str(p["day"]), "count": 10}, sort_keys=True).encode()
inventory = {key: {"sha256": hashlib.sha256(payload).hexdigest(), "bytes": len(payload), "day": json.loads(payload)["day"]} for key, payload in archive_bytes.items()}
sample = rng.sample(sorted(inventory), 5)
restored = {key: archive_bytes[key] for key in sample}
restored[sample[2]] += b"damage"

def verify_restore(key, payload, inventory):
    expected = inventory[key]
    if payload is None:
        return "MISSING"
    if len(payload) != expected["bytes"] or hashlib.sha256(payload).hexdigest() != expected["sha256"]:
        return "CHECKSUM MISMATCH"
    try:
        row = json.loads(payload)
        if not isinstance(row, dict) or type(row.get("schema_version")) is not int or row["schema_version"] != 1 or type(row.get("count")) is not int or row["count"] < 0:
            return "SCHEMA FAILURE"
        date.fromisoformat(row["day"])
        if row["day"] != expected["day"]:
            return "IDENTITY MISMATCH"
    except (ValueError, TypeError, KeyError, UnicodeDecodeError):
        return "CONSUMER FAILURE"
    return "ok"

results = [(key, verify_restore(key, restored.get(key), inventory)) for key in sample]
for key, verdict in results:
    print(key, verdict)
print("local drill findings:", sum(verdict != "ok" for _, verdict in results), "of", len(results))
# raw/events/ingest_date=2025-03-27/part-0000.json ok
# raw/events/ingest_date=2025-08-14/part-0000.json ok
# raw/events/ingest_date=2024-07-19/part-0000.json CHECKSUM MISMATCH
# raw/events/ingest_date=2024-06-17/part-0000.json ok
# raw/events/ingest_date=2023-04-22/part-0000.json ok
# local drill findings: 1 of 5

7. Write the lifecycle policy. Document filters, clocks, exceptions, economics, restore requirements, and deletion ownership.