Designing an Object-Storage Data Lake on S3, GCS, or ADLS
In plain terms
A data lake needs more than a place to upload files. A reader must know which objects form a complete dataset, what their schema means, and which source and code produced them. An operator must be able to retry a failed run, control access, and remove obsolete data without breaking active readers. Object storage supplies durable objects and APIs; the lake adds these contracts.
This article follows the ingestion series into storage layout. It covers zones, keys, partitions, publication, replay, file consolidation, security, and cost on S3, GCS, and ADLS. A metadata-only Python simulator makes request counts and failure cases visible. Its timings and prices are assumptions, not cloud measurements.
Objects, consistency, and metadata
An object has a key, bytes, size, and metadata. GET can read bytes or ranges; PUT, DELETE, and paginated prefix listing manage objects. List limits and continuation tokens depend on the API; the lab chooses 1,000 keys per page and counts one request even for an empty result. A flat namespace interprets slashes as key characters, while hierarchical namespace products provide actual directory operations.
An ETag is a service-defined entity validator, not a portable content hash. For example, an S3 multipart object’s ETag need not be an MD5 digest. Use a documented checksum algorithm for content integrity and a version ID or generation for object identity. The lab uses synthetic revision numbers and stores declared sizes rather than actual Parquet bytes. S3 object metadata.
Strong object read/list consistency does not make a sequence of writes a table transaction. Readers can observe several successfully written objects while later parts are still missing. Separate requests and paginated listings do not automatically share one snapshot. Cache behavior, permissions, replication, and notifications have their own guarantees. The design below relies on atomic conditional replacement of one publication object, plus immutable data parts; each cloud adapter must establish those semantics.
Zones and discoverable datasets
Landing preserves the permitted original payload and arrival metadata. Raw preserves source records and the identifiers required for replay. If converting JSON to Parquet loses fields or original formatting that must be retained, do not expire the only original copy merely because conversion succeeded. Staging holds typed and validated intermediate results; curated holds published tables; quarantine holds rejected input with a reason and owner. Quarantine can contain sensitive data and needs protection too.
Use separate buckets or containers where ownership, residency, encryption, retention, or access boundaries require them. Prefixes can be sufficient for other separations if the provider’s controls support them. Five zones do not require five buckets. Rebuildability depends on retained inputs, transformations, dependencies, and reference data, not just a zone name.
Keep an incoming delivery manifest as source evidence; it is distinct from the output snapshot selected for readers. Register datasets in a catalog with owner, schema and schema version, format, location or snapshot reference, freshness target, classification, and retention policy. A bucket does not enforce a table schema. Readers need engine-compatible table metadata and partition registration or discovery rules. A custom manifest is readable only by clients that implement its protocol; an ordinary SQL engine will not infer this contract from a JSON filename.
Keys and partition pruning
A useful raw key is raw/shop_db/orders/ingest_date=2026-03-15/run_id=r014/part-0000.parquet. It identifies source, table, arrival partition, and immutable run. For a curated table, choose partition dimensions from query patterns and data volume: event date is often useful, but it is not mandatory. Keep ingestion time and event time separately so late arrivals can be placed and traced correctly.
Prefix matching uses the leading characters, not the visual existence of folders. A filename beginning part-2026-03-15- can also be listed narrowly for that day. A date buried after a variable part number cannot be isolated with one common day prefix. Exercise 1 compares those cases. Catalog partitions, table manifests, and Parquet statistics can prune data beyond what keys alone reveal; key=value directories are a common engine convention, not a universal query feature.
Estimate bytes and files per partition along with predicate selectivity. One customer partition per tiny record can create excessive metadata and requests, but there is no universal limit of a few hundred partition values. A high-volume tenant or a bucketed layout can justify a different choice. Avoid secrets and personal identifiers in object names because paths can appear in logs. Distinguish immutable historical partition values from mutable current classifications.
Immutable runs and conditional publication
Write data under a fresh run ID with create-if-absent checks. A retry uses a new attempt ID or a documented idempotent recovery protocol. Verify file existence, counts, schemas, checksums, and business constraints before publication. A manifest lists the exact objects and their versions, sizes, schema, source snapshots, and code version. For a small dataset it can be the publication object itself; larger systems often publish an immutable manifest and switch a small pointer to it.
The writer first reads the current publication token. After preparing the new run, it replaces the publication object only if the token is still unchanged. This compare-and-swap condition prevents a stale writer from silently overwriting a newer commit. The token orders publication attempts, not source freshness: an old source snapshot can still win if prepared against the current token. Enforce any source-version or business-cutoff rule separately. If the condition fails, re-read and decide whether to recompute; do not simply retry unconditionally. S3 conditional writes illustrates the underlying precondition. Reader code loads one manifest once and follows only its references. Incomplete runs are not visible to that reader protocol.
A single atomic publication object is not a general transaction manager. Multiple tables, concurrent partition updates, ambiguous commit responses, and streaming checkpoints require more coordination. Iceberg and Delta provide fuller table protocols with different metadata and commit mechanisms; they are not interchangeable implementations of this small JSON example. A _SUCCESS marker works only with an agreed complete, immutable run and a reader that respects it.
Retain objects referenced by current snapshots, retained history, or active readers. A run absent from the current manifest may still be in use, and a day-old unfinished run may still be writing. Garbage collection needs reachability, writer state or leases, and a safe grace period; age alone is insufficient. Versioning, where supported and enabled, helps recover overwrites but does not replace tested backups, deletion policy, or reference tracking. Reproducible rows require pinned inputs, code, dependencies, configuration, reference data, and deterministic logic. Byte-identical files additionally depend on ordering, writer metadata, and compression behavior.
File consolidation and cost
Compaction consolidates small files; compression reduces their encoded byte size. They are different operations, although a rewrite can do both. Publish the replacement as a new run after checking row counts and content-level invariants, then retire old files only when safe. A small partition should not be padded to a nominal target size. Pick file and row-group sizes using engine scans, task parallelism, memory, and recovery time; hundreds of MB can be a starting experiment, not a requirement.
Exercise 2 compares equal 43.2-MB inputs: 1,440 small files and four 10.8-MB files. Eight simulated workers each have a fixed latency and independent bandwidth. The model schedules each object onto an available worker, so four files cannot benefit from eight-way division. Real shared bandwidth, range reads, caching, decoding, and throttling are outside the model. No speed ratio from it is a provider benchmark.
Cost includes average stored byte-time, retained versions, requests, retrieval and minimum-storage charges, network transfer, encryption-key operations, and compute. Measure by accountable dataset or owner as well as zone and bucket. Retention follows value, obligations, and recovery needs; low storage prices do not justify keeping everything. The lab uses invented unit prices and a steady-state 30-day retention window, with each new day’s data read twenty times. Its compact layout excludes conversion compute and temporary duplicate storage, so it is not a total project savings estimate.
Provider choices and security boundaries
Specify the storage product and namespace, not just the cloud name. S3 general-purpose buckets and S3 directory buckets have different features. RenameObject supports a single-object rename within an S3 Express One Zone directory bucket; it is not a transaction that swaps a multi-file table. S3 RenameObject.
GCS offers both flat and hierarchical namespaces. Its hierarchical namespace supports atomic folder rename but currently excludes Object Versioning. GCS also has an atomic object-move API; tool behavior may instead use copy and delete. An object move must not be confused with an atomic move of a simulated folder containing many objects. GCS hierarchical namespace; GCS object move.
ADLS Gen2 with hierarchical namespace supports directory operations and directory/file ACLs. Azure Blob versioning currently does not support accounts with hierarchical namespace enabled. Thus the lab’s combination of optional rename and always-retained versions is a conceptual model, not an ADLS or GCS HNS feature bundle. ADLS namespace; Azure Blob versioning.
Use private access for this internal lake, workload identities, least-privilege grants, encryption in transit and at rest, and explicit audit configuration. Deny raw mutation to normal writers except through the intended create-only path; separate publisher and cleanup privileges where practical. Keep logs and quarantine protected. Verify bucket/container/prefix or directory policy support, IAM/ACL interactions, key permissions, network paths, and restore behavior for the chosen product. Namespaces, conditional-write APIs, identity models, and cleanup tooling need adapters even when the publication design is shared.
Lab
Run the shared setup before each exercise in Python 3.12. It stores sizes, string payloads, synthetic revisions, and request counters in memory, not full data files. Atomic precondition checking is assumed within this single-process model. It does not implement cloud authorization, true network concurrency, durable recovery, version lifecycle, checksums over file bytes, or cloud directory APIs. Rename counters illustrate two strategies; they are not a provider billing model.
import hashlib, json, random
from collections import Counter, defaultdict
from datetime import date, timedelta
class Conflict(Exception):
pass
ANY = object()
class ObjectStore:
"""Single-process metadata model; versions and atomic preconditions are assumed."""
def __init__(self, hierarchical=False):
self.objects, self.versions = {}, defaultdict(list)
self.hierarchical, self.requests, self.sequence = hierarchical, Counter(), 0
def put(self, key, size, content="", expected=ANY):
if type(size) is not int or size < 0:
raise ValueError("size must be a nonnegative integer")
self.requests["PUT"] += 1
current = self.objects.get(key)
revision = current["revision"] if current else None
if expected is not ANY and revision != expected:
raise Conflict("precondition failed")
self.sequence += 1
obj = {"size": size, "revision": self.sequence, "content": content}
self.versions[key].append(dict(obj))
self.objects[key] = obj
return obj["revision"]
def get(self, key, version=None):
self.requests["GET"] += 1
if version is None:
return dict(self.objects[key])
for obj in self.versions.get(key, []):
if obj["revision"] == version:
return dict(obj)
raise KeyError((key, version))
def list(self, prefix, page_size=1000):
if type(page_size) is not int or page_size <= 0:
raise ValueError("page_size must be positive")
keys = sorted(k for k in self.objects if k.startswith(prefix))
for start in range(0, max(len(keys), 1), page_size):
self.requests["LIST"] += 1
yield from keys[start:start + page_size]
def delete(self, key):
self.requests["DELETE"] += 1
self.objects.pop(key, None)
def rename_prefix(self, old, new):
"""Count an idealized directory rename or a sequential copy/delete move."""
if not old or not new or not old.endswith("/") or not new.endswith("/") or old.startswith(new) or new.startswith(old):
raise ValueError("use distinct non-overlapping directory prefixes")
keys = sorted(k for k in self.objects if k.startswith(old))
if any(k.startswith(new) for k in self.objects):
raise Conflict("destination exists")
if self.hierarchical:
self.requests["RENAME"] += 1
else:
self.requests["COPY"] += len(keys)
self.requests["DELETE"] += len(keys)
for key in keys:
self.objects[new + key[len(old):]] = self.objects.pop(key)
MB = 1_000_000
rng = random.Random(0)
lake = ObjectStore()
first_day = date(2026, 3, 1)
for k in range(30):
day = first_day + timedelta(days=k)
for source in ("shop_db.orders", "events.checkout"):
for part in range(4):
lake.put(f"raw/{source}/ingest_date={day.isoformat()}/run_id=r{k:03d}/part-{part:04d}.parquet", rng.randint(2 * MB, 6 * MB))
print(len(lake.objects), "objects,", sum(o["size"] for o in lake.objects.values()) // MB, "MB,", dict(lake.requests))
print(next(iter(lake.list("raw/shop_db.orders/ingest_date=2026-03-15/"))))
# 240 objects, 977 MB, {'PUT': 240}
# raw/shop_db.orders/ingest_date=2026-03-15/run_id=r014/part-0000.parquet1. Keys and listing. Compare partitioned keys, filenames with a buried date, and filenames with a leading date.
Solution
def year_of_keys(layout, files_per_day):
keys = []
for k in range(365):
day = (date(2025, 3, 1) + timedelta(days=k)).isoformat()
for n in range(files_per_day):
keys.append(layout.format(day=day, n=n))
return keys
def list_calls(keys, prefix, page_size=1000):
if type(page_size) is not int or page_size <= 0:
raise ValueError("page_size must be positive")
matching = [k for k in keys if k.startswith(prefix)]
return len(matching), max(1, -(-len(matching) // page_size))
partitioned = year_of_keys("raw/events.checkout/ingest_date={day}/part-{n:04d}.parquet", 1440)
flat = year_of_keys("raw/events.checkout/part-{n:04d}-{day}.parquet", 1440)
print("one year of minute files:", len(partitioned), "objects")
print("read one day, partitioned keys: objects and LIST calls =", list_calls(partitioned, "raw/events.checkout/ingest_date=2025-09-10/"))
print("read one day, flat keys: objects and LIST calls =", list_calls(flat, "raw/events.checkout/"), "then filter by name in the client")
print("read one day, partitioned but with 4 daily files:", list_calls(year_of_keys("raw/events.checkout/ingest_date={day}/part-{n:04d}.parquet", 4), "raw/events.checkout/ingest_date=2025-09-10/"))
leading_date = year_of_keys("raw/events.checkout/part-{day}-{n:04d}.parquet", 1440)
print("date-leading filename also supports a narrow prefix:", list_calls(leading_date, "raw/events.checkout/part-2025-09-10-"))
# one year of minute files: 525600 objects
# read one day, partitioned keys: objects and LIST calls = (1440, 2)
# read one day, flat keys: objects and LIST calls = (525600, 526) then filter by name in the client
# read one day, partitioned but with 4 daily files: (4, 1)
# date-leading filename also supports a narrow prefix: (1440, 2)A broad scan returns 525,600 keys in 526 modeled pages. A useful day prefix returns 1,440 keys in two pages, whether that prefix is a directory convention or a date-leading filename. Four daily files fit in one page. This counts discovery requests under the stated strategy; metadata catalogs or a list of known keys can avoid a broad listing too.
2. Small files and consolidation. Model equal byte volumes with eight workers, then count metadata-only consolidation reads and writes.
Solution
import heapq
import math
def read_time_seconds(sizes, first_byte_ms=30, throughput_mb_s=100, concurrency=8):
"""Independent per-worker bandwidth; no shared network bottleneck or retries."""
if type(concurrency) is not int or concurrency <= 0:
raise ValueError("concurrency must be positive")
if not math.isfinite(first_byte_ms) or first_byte_ms < 0 or not math.isfinite(throughput_mb_s) or throughput_mb_s <= 0:
raise ValueError("invalid latency or throughput")
workers = [0.0] * concurrency
for size in sizes:
if type(size) is not int or size < 0:
raise ValueError("invalid size")
finish = heapq.heappop(workers) + first_byte_ms / 1000 + size / (throughput_mb_s * MB)
heapq.heappush(workers, finish)
return max(workers)
minute_files = [30_000] * 1440
compacted = [sum(minute_files) // 4] * 4
assert sum(compacted) == sum(minute_files)
print(f"minute files: {sum(minute_files) / MB:.1f} MB, {read_time_seconds(minute_files):.3f} s, {len(minute_files)} GETs")
print(f"compacted: {sum(compacted) / MB:.1f} MB, {read_time_seconds(compacted):.3f} s, {len(compacted)} GETs")
print(f"model speed ratio: {read_time_seconds(minute_files) / read_time_seconds(compacted):.1f}")
staging = ObjectStore()
for n, size in enumerate(minute_files):
staging.put(f"landing/events.checkout/2026-03-15/minute-{n:04d}.json", size)
source_keys = list(staging.list("landing/events.checkout/2026-03-15/"))
staging.requests.clear()
total_bytes = sum(staging.get(key)["size"] for key in source_keys)
for part in range(4):
staging.put(f"staging/events.checkout/run_id=compact_20260315/part-{part:04d}.parquet", total_bytes // 4, expected=None)
print("metadata-only compaction requests after input discovery:", dict(staging.requests))
print("landing objects retained:", len(list(staging.list("landing/"))))
# minute files: 43.2 MB, 5.454 s, 1440 GETs
# compacted: 43.2 MB, 0.138 s, 4 GETs
# model speed ratio: 39.5
# metadata-only compaction requests after input discovery: {'GET': 1440, 'PUT': 4}
# landing objects retained: 1440The model gives 5.454 seconds for minute files and 0.138 seconds for four files, a 39.5-fold ratio under independent per-worker bandwidth. These are scheduled work estimates, not elapsed measurements. The consolidation example reads all 1,440 input metadata records and writes four outputs with the same total declared size. It leaves landing objects intact but does not transform JSON into Parquet or validate record equivalence.
3. Conditional publication. Prepare a run, crash another, commit a replacement conditionally, and reject a stale publisher.
Solution
def prepare_run(store, table, run_id, parts, crash_after=None):
if type(parts) is not int or parts < 0:
raise ValueError("parts must be nonnegative")
if crash_after is not None and (type(crash_after) is not int or not 0 <= crash_after <= parts):
raise ValueError("invalid crash position")
prefix = f"curated/{table}/run_id={run_id}/"
store.put(prefix + "_intent.json", 2, content="{}", expected=None)
written = []
for n in range(parts):
if n == crash_after:
return None
key = prefix + f"part-{n:04d}.parquet"
revision = store.put(key, 5 * MB, content=run_id, expected=None)
written.append({"key": key, "revision": revision, "size": 5 * MB})
if crash_after == parts:
return None
return {"table": table, "run_id": run_id, "objects": written}
def read_manifest(store, manifest):
prefix = f"curated/{manifest['table']}/run_id={manifest['run_id']}/"
keys = [part["key"] for part in manifest["objects"]]
if len(keys) != len(set(keys)):
raise ValueError("duplicate manifest part")
if any(not key.startswith(prefix) or not key.endswith(".parquet") for key in keys):
raise ValueError("part outside declared run")
for part in manifest["objects"]:
obj = store.get(part["key"])
if (obj["revision"], obj["size"]) != (part["revision"], part["size"]):
raise ValueError("published part changed")
return manifest["run_id"], len(manifest["objects"])
def commit_run(store, table, manifest, expected):
if manifest is None:
raise ValueError("incomplete run")
if manifest["table"] != table:
raise ValueError("manifest table differs")
if expected is not None and (type(expected) is not int or expected <= 0):
raise ValueError("publication requires an explicit revision or absence")
read_manifest(store, manifest)
content = json.dumps(manifest, sort_keys=True)
return store.put(f"curated/{table}/_manifest.json", len(content.encode()), content, expected=expected)
def read_published(store, table):
manifest = json.loads(store.get(f"curated/{table}/_manifest.json")["content"])
if manifest["table"] != table:
raise ValueError("manifest table differs")
return read_manifest(store, manifest)
store = ObjectStore()
first_manifest = prepare_run(store, "daily_revenue", "r001", 4)
first_revision = commit_run(store, "daily_revenue", first_manifest, expected=None)
failed = prepare_run(store, "daily_revenue", "r002", 4, crash_after=2)
print("after a partial run:", read_published(store, "daily_revenue"), "incomplete:", failed is None)
third_manifest = prepare_run(store, "daily_revenue", "r003", 4)
third_revision = commit_run(store, "daily_revenue", third_manifest, expected=first_revision)
print("after next commit:", read_published(store, "daily_revenue"))
print("old pinned manifest remains readable:", read_manifest(store, first_manifest))
print("parts exposed by a broad listing:", sum(k.endswith(".parquet") for k in store.list("curated/daily_revenue/")))
stale = prepare_run(store, "daily_revenue", "r004", 4)
try:
commit_run(store, "daily_revenue", stale, expected=first_revision)
except Conflict:
print("stale publisher rejected; current:", read_published(store, "daily_revenue"))
for hierarchical in (False, True):
sample = ObjectStore(hierarchical=hierarchical)
for n in range(200):
sample.put(f"tmp/part-{n:04d}.parquet", 5 * MB)
sample.requests.clear()
sample.rename_prefix("tmp/", "current/")
print("idealized move", "hierarchical" if hierarchical else "flat", dict(sample.requests))
# after a partial run: ('r001', 4) incomplete: True
# after next commit: ('r003', 4)
# old pinned manifest remains readable: ('r001', 4)
# parts exposed by a broad listing: 10
# stale publisher rejected; current: ('r003', 4)
# idealized move flat {'COPY': 200, 'DELETE': 200}
# idealized move hierarchical {'RENAME': 1}The incomplete run leaves parts and an intent record, but the publication still names r001. A successful conditional replacement selects r003. A reader that already pinned r001 can still read it because its parts remain. The old publication token cannot replace r003. The reader fetches every named part and checks revision and size; it fails on missing or overwritten data instead of silently returning a part count. The manifest also names its table and run; duplicate part keys, references outside that run, and a different publication table are rejected. Publication requires an explicit revision or an absence condition, so the store’s unconditional-write sentinel cannot bypass it. This trusted-writer toy validates metadata, not schemas, complete source coverage, or content. It does not protect against an adversarial writer changing data between validation and commit; immutable-part permissions remain part of the contract. _intent.json reserves a run name; it is not a lease or a completion signal. The idealized move counts exclude discovery and do not model intermediate failures or provider charges.
4. Version references and replay. Overwrite a raw key, recover its old version, and fingerprint pinned input metadata and payload.
Solution
key = "raw/shop_db.orders/ingest_date=2026-03-15/run_id=r014/part-0000.parquet"
original = lake.get(key)
original_revision = original["revision"]
lake.put(key, 3 * MB, content="rerun with a bug fix")
print("overwrite changes revision:", original_revision != lake.get(key)["revision"], "retained versions:", len(lake.versions[key]))
print("original version recoverable:", lake.get(key, original_revision) == original)
print("new key for the rerun:", key.replace("run_id=r014/", "run_id=r014_rerun/"))
def fingerprint(code_version, refs):
inputs = []
for key, revision in sorted(refs):
obj = lake.get(key, revision)
inputs.append((key, revision, obj["size"], obj["content"]))
payload = json.dumps({"code": code_version, "inputs": inputs}, sort_keys=True)
return hashlib.sha256(payload.encode()).hexdigest()
pinned = [(key, original_revision)]
current = [(key, lake.get(key)["revision"])]
print("same code and pinned inputs:", fingerprint("v1", pinned) == fingerprint("v1", pinned))
print("overwrite detected by refreshed inputs:", fingerprint("v1", pinned) != fingerprint("v1", current))
print("code change changes provenance:", fingerprint("v1", pinned) != fingerprint("v2", pinned))
# overwrite changes revision: True retained versions: 2
# original version recoverable: True
# new key for the rerun: raw/shop_db.orders/ingest_date=2026-03-15/run_id=r014_rerun/part-0000.parquet
# same code and pinned inputs: True
# overwrite detected by refreshed inputs: True
# code change changes provenance: TrueThe retained original is recoverable by its synthetic revision. A refreshed input reference changes the fingerprint after an overwrite, unlike hashing filenames alone. The code label also changes the provenance fingerprint. This hashes a serialized description of the toy inputs; it does not execute a transformation or prove byte-identical Parquet output. Pin actual source versions and execution dependencies when testing real replay.
5. Partition tradeoffs. Count candidate objects for three layouts, without using file statistics or a customer-to-country lookup.
Solution
def layout_stats(layout, customers=2000, days=30, countries=3):
if type(countries) is not int or not 1 <= countries <= 3:
raise ValueError("countries must be between 1 and 3")
keys = set()
for d in range(days):
day = (first_day + timedelta(days=d)).isoformat()
for c in range(customers):
keys.add(layout.format(day=day, customer=f"C{c}", country=["US", "KR", "DE"]))
return keys
layouts = {
"by customer": "raw/orders/customer_id={customer}/{day}.parquet",
"by date": "raw/orders/ingest_date={day}/part.parquet",
"by date and country": "raw/orders/ingest_date={day}/country={country}/part.parquet",
}
print(f"{'layout':22} {'objects':>8} {'one day, all':>13} {'one customer':>13} {'one day, KR':>12}")
for name, layout in layouts.items():
keys = layout_stats(layout)
one_day = sum(k.find("2026-03-15") >= 0 for k in keys)
one_customer = sum("customer_id=C7/" in k or "customer_id=C7." in k for k in keys) or len(keys)
one_day_kr = sum("2026-03-15" in k and ("country=KR" in k or "country=" not in k) for k in keys)
print(f"{name:22} {len(keys):8} {one_day:13} {one_customer:13} {one_day_kr:12}")
print("candidate objects for 'one customer' under the date layouts includes every object when no customer-to-country mapping or file statistics are available")
# layout objects one day, all one customer one day, KR
# by customer 60000 2000 30 2000
# by date 30 1 30 1
# by date and country 90 3 90 1
# candidate objects for 'one customer' under the date layouts includes every object when no customer-to-country mapping or file statistics are availableThe counts assume one file per unique generated partition key. A customer layout creates 60,000 objects; date-only creates 30; date-country creates 90. Files have unequal implied row counts, so fewer objects does not automatically mean fewer bytes. The candidate counts assume no external customer-country mapping, no file statistics, and no clustering. The set deduplicates partition names; it does not repeatedly write and overwrite a file for each customer.
6. A bounded cost model. Separate storage and request cost for equal retained bytes, using illustrative prices.
Solution
PRICE = {"storage_gb_month": 0.023, "put_per_1000": 0.005, "list_per_1000": 0.005, "get_per_1000": 0.0004}
def monthly_cost(puts, average_gb, gets, lists):
if any(v < 0 for v in (puts, average_gb, gets, lists)):
raise ValueError("usage cannot be negative")
storage = average_gb * PRICE["storage_gb_month"]
requests = (puts * PRICE["put_per_1000"] + gets * PRICE["get_per_1000"] + lists * PRICE["list_per_1000"]) / 1000
return {"storage": storage, "requests": requests, "total": storage + requests}
# Steady-state 30-day retention; each incoming day's data is read 20 times.
average_gb = 1440 * 30_000 * 30 / 1_000_000_000
minute = monthly_cost(1440 * 30, average_gb, 1440 * 30 * 20, 30 * 20 * 2)
compact = monthly_cost(4 * 30, average_gb, 4 * 30 * 20, 30 * 20)
for name, cost in (("minute", minute), ("compact", compact)):
print(name, " ".join(f"{key}={value:.4f}" for key, value in cost.items()))
print(f"steady-state modeled cost ratio: {minute['total'] / compact['total']:.1f}")
print("both layouts store", average_gb, "average GB; conversion compute and temporary copies excluded")
# minute storage=0.0298 requests=0.5676 total=0.5974
# compact storage=0.0298 requests=0.0046 total=0.0344
# steady-state modeled cost ratio: 17.4
# both layouts store 1.296 average GB; conversion compute and temporary copies excludedBoth layouts retain an average 1.296 decimal GB. At these invented prices the totals are 0.5974 and 0.0344 currency units per month, about 17.4 times apart. This workload reads each daily partition twenty times; it does not scan the entire retained month twenty times every day. The compact case assumes data arrives in compact form and excludes conversion, temporary duplicates, retries, and metadata overhead. A real compaction decision adds those costs and uses current region, tier, and account prices.
7. Write the lake standard. Specify zone keys, catalog registration, publication, recovery, access, and safe cleanup for one chosen product.
Solution
Use landing/shop_db/arrival_date=2026-03-15/batch=b1/input.json, raw/shop_db/orders/ingest_date=2026-03-15/run_id=r1/part-0000.parquet, staging/orders/run_id=s1/part-0000.parquet, curated/orders/run_id=c1/part-0000.parquet, and quarantine/shop_db/batch=b1/reason.json as concrete examples. Assign owners and readers, register schemas and source versions, and document the permitted payload and retention in each zone. Require fresh run names, conditional creation and publication, and immutable referenced parts. Define recovery after a lost commit response and pin reader snapshots. Evaluate file size and compaction frequency from measured scans and arrival rates. Cleanup must check active writers/readers and every retained snapshot, not merely whether a run is older than a day. Choose a specific bucket/account type and verify conditional writes, version protection, namespace, IAM, notifications, and restore tests. Retention dates and target file sizes are explicit workload decisions, not universal constants.
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.