Designing End-to-End Data Architecture

Architecture records the decisions that shape how a system can change: where data is stored, how it moves, what each stage promises, and who owns the result. A data architecture connects source systems to useful outputs while accounting for failures, access, cost, and recovery. Existing tools and team skills constrain the choices; consumer requirements give those choices a purpose.

The preceding cloud infrastructure article mapped compute, storage, networks, and identity. Here we connect those resources to consumer contracts and data flow through a fictional subscription shop. We examine three growth stages, then build a small four-layer example. Four layers are a design used here, not a requirement for every platform. The optional code lab assumes basic Python, SQL, transactions, and validation.

Start with the consumers and their decisions

Ask what someone will do with the output, when it must be ready, and what happens if it is late or wrong. Then check whether the sources can support that request. For this fictional shop, the following are assumed requirements to confirm with each owner. They are not industry-wide defaults.

Consumer	Output and timing	Design consequence
Finance	Daily revenue by date and currency; ready by 06:00 UTC; unexplained differences reviewed before close	Retain relevant source evidence; agree reconciliation and correction rules
Executives	Defined KPIs by 08:00 UTC	An owned metric definition and visible freshness status
Product analysts	Daily event detail and funnels initially	Governed access to detailed, cleaned events
Fraud operations	Daily manual review initially; a possible later 60-second alert requirement	Document the accepted interim process and the trigger for a faster path
Marketing	Daily customer segments sent to an email service	Reverse ETL: send modeled results to an operational tool
ML team	Daily training data using only information available at the prediction time	Retain relevant history and test for future-information leakage; serving latency is a separate requirement

This table supports a batch-first design for this shop. Another product may need continuous processing from its first day. Shared revenue definitions should have an owner and a reusable implementation; that does not require every consumer to read one physical table. A metric can be materialized in several places if its definition, version, and reconciliation remain controlled.

For the marketing export, define the destination record key and which system owns each field. Updating a segment is not the same operation as sending a campaign: retries must not trigger repeated messages. Specify what happens when a person leaves the segment, and verify destination state rather than treating an accepted API request as proof that every row was applied.

Four logical layers, with explicit promises

Sources → landing/raw → staging/cleaned → core/modeled → marts/serving → consumers
          source evidence   usable records     shared meaning   consumer outputs

Across all layers: ownership, access, scheduling, quality, lineage, recovery, cost

These are logical responsibilities. They can be separate schemas in one database, files and tables in different systems, or views without an extra stored copy. Bronze, silver, and gold in a medallion design describe related refinement stages, but their boundaries do not map uniquely to these four layers. Add a boundary when it gives you a distinct contract, owner, or operational need.

Landing / raw: preserve the evidence you need

Keep received files unchanged when byte-for-byte evidence matters. For a database extract or API response, document what was captured, the source position or extraction time, and any serialization changes. Converting a CSV into Parquet changes its bytes and may change its interpretation; it is not the same as retaining the original file. An event system may preserve ordering within a partition without providing a global arrival order.

Preserved inputs let you rerun a corrected transformation, provided the required data, reference data, and code versions remain available. Define retention by purpose and dataset, with controlled deletion and access. Append-only ingestion prevents routine overwrites; it does not forbid an authorized deletion process. Budget for storage, retrieval, and replay time together.

Staging / cleaned: make source records usable

Parse types, standardize names, handle duplicate deliveries, and quarantine invalid records with reasons. Define the grain, meaning what one row represents: one order, one order version, or one event are different grains. Removing duplicate deliveries must not remove legitimate updates. Convert timestamps only when the source timezone is known, and preserve any business-calendar context needed later.

Core / modeled: agree on shared meaning

Connect source records into customers, orders, payments, and other business entities or events. Define which statuses count toward a metric, how refunds are treated, and who owns each definition. When a report needs the customer country at order time, join against dated customer history; a join to the current customer table cannot reconstruct that history. The data modeling article develops these choices.

Marts / serving: shape the output for its reader

A finance mart might contain one row per day and currency; an analyst may need event-level detail; an application may need a keyed lookup. Marts need not all be wide or aggregated. Rebuildable outputs still need a publication plan: build and validate a replacement, then switch readers to it, keeping the previous version if recovery requires it. Dropping a live mart before rebuilding it creates an outage.

A layer contract should name the producer, grain and keys, schema and units, event and arrival times, supported updates/deletes, quality policy, and consumer notification process. For the finance output, define a complete date/currency population and an owner who can approve corrections. A new nullable column can be compatible for named-column readers yet break SELECT * exports; metric changes can be incompatible without any schema change. Version the contract and test affected consumers before switching them.

For this example, derived tables read lower layers and never write into their inputs. The resulting dependency graph has no cycles and can be built in order. This is a useful local rule, not a ban on governed analyst access to staging or every same-layer dependency in other architectures. Shared business definitions belong in a controlled model or semantic layer; consumer-specific presentation can remain in serving.

Design operation alongside data flow

The data plane stores and processes records. The control plane configures and coordinates that work: scheduling, permissions, metadata, and operating policy. Quality checks execute against data, while their rules and results inform control decisions; the boundary is conceptual rather than a separate product requirement.

• Orchestration: wait for required inputs, run dependencies in order, retry safely, and stop publication when a required check fails. A retry should not duplicate yesterday’s output.
• Quality and observability: measure freshness, volume, and agreed invariants. Record an owner and a response for each alert.
• Catalog and lineage: record what tables mean and which inputs produced them, including the relevant run and code version.
• Access and environments: separate production permissions, keep credentials out of source code, and use suitable masked or synthetic test data. Promote the same code artifact with environment-specific configuration.
• Recovery and cost: define how to restore service, test that procedure, and attribute storage and processing spend. A cron job can be adequate if dependencies, failure handling, and ownership are explicit.

Choose storage and ingestion from those requirements

A warehouse provides a database designed for analytical workloads. A lake stores datasets as files, often in object storage. A lakehouse combines lake storage with table-management and query capabilities. Table formats such as Iceberg add metadata and transaction mechanisms beyond individual files, but engine compatibility and supported operations still need checking. Warehouses can also work with open tables; proprietary storage is not their defining property. Separating storage and compute can help independent scaling, but it does not by itself guarantee portability.

ETL transforms data before loading the chosen analytical destination; ELT loads it there before transforming it. The distinction is relative to that destination, and one architecture can use both. Choose the transformation boundary from sensitive-data handling, available processing capacity, replay needs, and source/destination constraints. An open table format alone does not guarantee multi-table atomicity or interoperability; check the Iceberg specification and the engines and catalog used for the intended operations.

Batch processes a bounded input; micro-batching repeats that work in small increments; streaming processes an ongoing input. An hourly scheduled job is still batch, and a streaming engine may itself use micro-batches. Compare required latency, source delivery, operating effort, and measured cost. If batch and streaming outputs coexist, agree event-time windows, duplicate handling, late corrections, and metric versions. Writing both to one core layer alone does not make their numbers equal.

A snapshot captures state at a point in time. Comparing two complete snapshots by key can identify rows no longer present, but cannot recover every intermediate event: an order created and deleted between snapshots is absent from both. A watermark extracts changes after a stored position, such as an update timestamp; ties, late updates, and deletes require a policy. Log-based change data capture (CDC) can capture updates and deletes, but depends on capture scope, retained logs, and recovery from the saved position. Plan for redelivery and the ordering guarantees of the specific source. PostgreSQL’s logical decoding documentation explains snapshot alignment and possible repeated delivery after a crash.

For timestamp-based incremental extraction, choose a fixed upper bound for each run and advance the saved checkpoint only after the destination commits successfully. A key can break pagination ties within a stable extraction, but it does not catch rows committed later with an older timestamp. An overlap window plus idempotent writes can reduce that risk only within a justified lateness bound. Hard deletes need a tombstone, a change feed, or a separate comparison. This extraction watermark is different from a streaming event-time watermark, which tracks progress for window processing.

Pull means the platform requests data; push means a producer sends it. Either may be used for state or events. A receiving broker has finite capacity and can fail or reject requests: design acknowledgements, buffering, retries, and backpressure, which slows producers when receivers cannot keep up.

Write measurable non-functional requirements

Non-functional requirements describe how well the system must operate. An SLO is a service-level objective, such as a target proportion of on-time daily runs. RPO is the maximum tolerable data-loss interval; RTO is the target time to restore service. The following are illustrative design inputs, not legal retention rules or measured facts about a real company.

Requirement	Example to agree and test
Freshness	At least 99% of daily finance runs ready by 06:00 UTC over a calendar month
Accuracy and completeness	Reconcile order counts and amounts by date, currency, and agreed status; investigate every unexplained difference before close
Volume and growth	Assume 40 million events and 200 GB raw per day at stage 2; measure peak rates and query concurrency
Retention	Assign a period to each dataset and purpose; track deletion through raw, derived tables, exports, and backups
Recovery	Example: RPO 24 hours, RTO 4 hours; confirm consumers accept this and demonstrate restoration from retained inputs
Privacy	Classify personal fields, define authorized roles, and specify test-data handling
Availability	Agree the acceptable success rate during business hours and the incident response owner
Cost	Set a monthly budget and an 80% alert; include ingestion, queries, retrieval, and recovery tests

Define the measurement population and start/end points. With one scheduled finance run per day, a 30-day month has only 30 observations: one missed deadline gives 29/30 = 96.67%, so a 99% monthly target permits no misses in that month. Decide in advance how source delays, planned exclusions, reruns, and incomplete but timely outputs count. Retrying a failed run must not add an extra success to the denominator. Report correctness/completeness separately as well as the ready-by deadline.

Check requirements together. A four-hour recovery target is not credible if restoring cold data alone takes longer. A second region may help some failure scenarios, but its need follows from the failure model, recovery objectives, and budget together, not merely the dashboard’s opening hours. Unknown values need an owner and a validation date.

Record decisions, including their limits

An architecture decision record (ADR) preserves the context, choice, alternatives, and consequences. Record assumptions separately from measured costs, and write the condition for revisiting a decision. Keep proposed and accepted decisions distinct; when an accepted decision changes, link the replacement and retain the earlier rationale. For example:

ADR-0003: Preserve received vendor files separately from parsed tables
Status: proposed for the fictional shop
Context: We need to diagnose parsing errors and replay retained deliveries.
Decision: Keep received CSV files in a restricted landing area, identified by
          source, arrival time, and delivery ID. Build typed tables separately.
Alternatives: Warehouse landing tables; keeping only parsed Parquet.
Trade-off: Original files retain parsing evidence but add storage and access work.
Retention: Agree dataset-specific periods and controlled deletion before launch.
Revisit: Measured retrieval cost, restore time, or source contracts change.

A fictional shop grows in three stages

The shop has 200,000 customers, an application database, payment and support APIs, event files, and a logistics CSV feed. These are illustrative inputs, not a report of the author’s employer. The table above describes its initial consumers.

Stage 1: daily processing with explicit ownership

Sources → scheduled ingestion → retained landing files
                                    ↓
                         staging → core → marts → consumers
                         SQL on DuckDB or a small warehouse
Operations: dependency scheduling, validation, restricted access, recovery procedure

Use a small deployment if measured runtimes and concurrency fit it. The largest table being 30 GB is not enough evidence by itself: joins, memory use, disk spill, and query load also matter. The team accepts daily fraud review for now. A confirmed 60-second requirement would change the design immediately; it should not be dismissed merely to keep the diagram simple.

Stage 2: grow where measured constraints appear

Assume two years later the shop has 2 million customers and 40 million events per day. A four-hour database export misses its agreed window; concurrent analyst queries miss their latency target. Evaluate CDC for the affected tables and a warehouse or lakehouse for the analytical workload. If several engines need shared tables, test an open table format with those engines. Keep original landing evidence separately where required. Expand cataloging, access controls, deployment automation, and cost tracking from their stage-1 baseline. No fixed cost ratio justifies these changes; compare representative workloads.

Stage 3: add the path that the latency requirement needs

Events → broker → stream processor → low-latency fraud store → fraud tool
                       ↓
             retained events/results → shared models → daily reports

Allocate the 60-second end-to-end target across the critical path. A provisional budget might allow 20 seconds for source delivery, 15 for processing, 10 for serving visibility, and 5 for alert delivery, leaving 10 seconds of margin. These are design allowances to validate, not measured percentiles that can simply be added. Measure actual end-to-end delay and account for source clock quality, queueing, and late events. If the source delivers every five minutes, a faster processor alone cannot meet 60 seconds.

Now suppose fraud operations needs signals within 60 seconds of the source event. Measure that delay end to end, including ingestion and the fraud store, not just processor runtime. Assign an operator, a replay procedure, stable event identifiers for deduplication, and rules for late events. Provisional alerts and finalized daily reports may differ until late data is reconciled. The useful continuity across stages is ownership and meaning; physical storage or layer boundaries may change when a documented requirement warrants it.

Lab: make the layer contracts observable

Use the fictional shop for six exercises. The standalone code generates synthetic input and uses Python’s standard library plus SQLite window functions. CSV paths represent order dates, not arrival dates; customers contain current country only. The lab demonstrates layer dependencies, exact integer monetary units, reruns, and a local publication gate. Durable ingestion, historical joins, multi-user concurrency, production permissions, and cloud deployment remain outside this lab.

1. Write the consumer table. Include output grain, deadline, owner, and action on failure. State which requirements are confirmed and which are assumptions; do not force a predetermined number of streaming consumers.

2. Write the non-functional requirements page. Make timing, recovery, and cost targets measurable. For qualitative requirements such as privacy classification, name the rule and owner rather than inventing a number. Check that retention and recovery requirements can be met together.

3. Draw stage 1 and run the four-layer example. This generated feed uses USD or EUR, non-negative amounts with one to nine digits before the decimal point and exactly two after it, known statuses, and valid UTC timestamps. Amounts become integer cents, and currencies are never added together. Identical payloads with the same order ID are redeliveries; a different payload for that ID stops ingestion rather than silently selecting an update. The timestamp replacement only reformats these generated UTC strings; it is not a general timestamp validator.

import csv, random, re, sqlite3, tempfile
from pathlib import Path
from decimal import Decimal

# Synthetic order-date files, including one identical redelivery.
root = Path(tempfile.mkdtemp())
raw_dir = root / "raw" / "orders"
rng = random.Random(0)
days = ["2026-03-12", "2026-03-13", "2026-03-14"]
for day in days:
    part = raw_dir / f"date={day}" / "part-0.csv"
    part.parent.mkdir(parents=True)
    with part.open("w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["order_id", "customer_id", "ordered_at", "status", "amount", "currency"])
        for i in range(1, 301):
            writer.writerow([f"O{day[-2:]}{i:03d}", f"C{rng.randint(1, 210)}",
                             f"{day}T{rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}:00Z",
                             rng.choices(["paid", "captured", "refunded"], weights=[80, 10, 10])[0],
                             "N/A" if i % 100 == 0 else f"{rng.uniform(5, 100):.2f}",
                             rng.choice(["USD", "EUR"])])
redelivered = raw_dir / "date=2026-03-13" / "part-1.csv"
redelivered.write_bytes((raw_dir / "date=2026-03-13" / "part-0.csv").read_bytes())

con = sqlite3.connect(":memory:", cached_statements=0)
# This feed requires a non-negative amount with exactly two decimal places.
def valid_amount(value):
    return int(isinstance(value, str) and re.fullmatch(r"[0-9]{1,9}[.][0-9]{2}", value) is not None)

def amount_cents(value):
    if not valid_amount(value):
        return None
    whole, fraction = value.split(".")
    return int(whole) * 100 + int(fraction)

con.create_function("valid_amount", 1, valid_amount)
con.create_function("amount_cents", 1, amount_cents)
con.execute("CREATE TABLE raw_orders (order_id TEXT, customer_id TEXT, ordered_at TEXT, status TEXT, "
            "amount TEXT, currency TEXT, _source_file TEXT)")
FIELDS = ["order_id", "customer_id", "ordered_at", "status", "amount", "currency"]
seen_orders = {}
def accept_delivery(row):
    payload = tuple(row[field] for field in FIELDS)
    key = row["order_id"]
    if key in seen_orders and seen_orders[key] != payload:
        raise ValueError(f"conflicting delivery for {key}; a version policy is required")
    seen_orders[key] = payload
    return payload

for part in sorted(raw_dir.glob("date=*/part-*.csv")):
    with part.open(newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        if reader.fieldnames != FIELDS:
            raise ValueError("unexpected CSV header")
        con.executemany("INSERT INTO raw_orders VALUES (?, ?, ?, ?, ?, ?, ?)",
                        (accept_delivery(r) + (str(part.relative_to(root)),) for r in reader))
con.execute("CREATE TABLE raw_customers (customer_id TEXT, country TEXT)")
con.executemany("INSERT INTO raw_customers VALUES (?, ?)",
                ((f"C{i}", rng.choice(["KR", "JP", "US"])) for i in range(1, 201)))
print(con.execute("SELECT count(*), count(DISTINCT order_id) FROM raw_orders").fetchone())
# (1200, 900)
ORDER = ["raw", "staging", "core", "marts"]
# Derived tables are SELECT statements over lower layers.
LAYERS = {
    "staging": {
        "stg_orders": """
            SELECT order_id, customer_id, replace(replace(ordered_at, 'T', ' '), 'Z', '') AS ordered_at_utc,
                   status, amount_cents(amount) AS amount_cents, currency
            FROM (SELECT *, row_number() OVER (PARTITION BY order_id ORDER BY _source_file) AS rn FROM raw_orders)
            WHERE rn = 1 AND valid_amount(amount) = 1""",
        "stg_orders_quarantine": """
            SELECT DISTINCT order_id, amount, 'expected up to nine digits and two decimals' AS reason
            FROM raw_orders WHERE valid_amount(amount) = 0""",
        "stg_customers": "SELECT customer_id, country FROM raw_customers",
    },
    "core": {
        "core_orders": """
            SELECT o.order_id, o.customer_id, c.country, c.customer_id IS NOT NULL AS customer_matched, o.ordered_at_utc, o.status,
                   o.status IN ('paid', 'captured') AS is_revenue_eligible, o.amount_cents, o.currency
            FROM stg_orders o LEFT JOIN stg_customers c USING (customer_id)""",
    },
    "marts": {
        "mart_daily_revenue": """
            SELECT substr(ordered_at_utc, 1, 10) AS day, currency, count(*) AS orders, sum(amount_cents) AS revenue_cents
            FROM core_orders WHERE is_revenue_eligible GROUP BY 1, 2""",
        "mart_orphan_orders": "SELECT * FROM core_orders WHERE customer_matched = 0",
    },
}
def query_violations(layer, name, sql, registry):
    ranks = {"raw_orders": 0, "raw_customers": 0}
    ranks.update({table: ORDER.index(group)
                  for group, tables in registry.items() for table in tables})
    reads = set()

    def collect_reads(action, table, column, database, trigger):
        if action == sqlite3.SQLITE_READ:
            reads.add(table)
        return sqlite3.SQLITE_OK

    con.set_authorizer(collect_reads)
    try:
        # Prepare the SELECT without executing it; SQLite reports actual table reads.
        con.execute("EXPLAIN " + sql).fetchall()
    finally:
        con.set_authorizer(None)
    return [f"{name} reads {table}" for table in sorted(reads)
            if table not in ranks or ranks[table] >= ORDER.index(layer)]


def one_way_violations(layers):
    return [problem for layer, tables in layers.items() for name, sql in tables.items()
            for problem in query_violations(layer, name, sql, layers)]

con.commit()

def build_layers():
    with con:
        con.execute("BEGIN")
        for layer, tables in LAYERS.items():
            for name, sql in tables.items():
                problems = query_violations(layer, name, sql, LAYERS)
                if problems:
                    raise ValueError(problems)
                # Trusted identifiers from this local registry, not external input.
                con.execute(f"DROP TABLE IF EXISTS {name}")
                con.execute(f"CREATE TABLE {name} AS {sql}")

build_layers()
for layer, tables in LAYERS.items():
    for name in tables:
        print(f"{layer:8} {name:24} {con.execute(f'SELECT count(*) FROM {name}').fetchone()[0]:5d} rows")
# staging  stg_orders                 891 rows
# staging  stg_orders_quarantine        9 rows
# staging  stg_customers              200 rows
# core     core_orders                891 rows
# marts    mart_daily_revenue           6 rows
# marts    mart_orphan_orders          47 rows
for row in con.execute("SELECT * FROM mart_daily_revenue ORDER BY day, currency"):
    print(row)
# ('2026-03-12', 'EUR', 123, 610899)
# ('2026-03-12', 'USD', 137, 738886)
# ('2026-03-13', 'EUR', 118, 571830)
# ('2026-03-13', 'USD', 144, 770743)
# ('2026-03-14', 'EUR', 130, 707867)
# ('2026-03-14', 'USD', 129, 675011)

print(one_way_violations(LAYERS))
# []
LAYERS["core"]["core_customer_value"] = "SELECT currency, sum(revenue_cents) FROM mart_daily_revenue GROUP BY 1"
print(one_way_violations(LAYERS))
# ['core_customer_value reads mart_daily_revenue']
del LAYERS["core"]["core_customer_value"]

# Reconcile the generated source independently in Python, preserving currencies.
expected = {}
for payload in seen_orders.values():
    order_id, customer_id, stamp, status, amount, currency = payload
    if amount == "N/A" or status not in {"paid", "captured"}:
        continue
    key = (stamp[:10], currency)
    count, total = expected.get(key, (0, 0))
    expected[key] = (count + 1, total + int(Decimal(amount) * 100))
actual = {(day, currency): (count, cents) for day, currency, count, cents
          in con.execute("SELECT * FROM mart_daily_revenue")}
assert actual == expected
assert con.execute("SELECT count(*) FROM core_orders").fetchone()[0] == 891
baseline = con.execute("SELECT * FROM mart_daily_revenue ORDER BY day, currency").fetchall()
build_layers()
assert con.execute("SELECT * FROM mart_daily_revenue ORDER BY day, currency").fetchall() == baseline
print("source reconciliation and rerun: passed")
# source reconciliation and rerun: passed

con.execute("CREATE TABLE published_revenue AS SELECT * FROM mart_daily_revenue WHERE 0")
con.commit()

def publish_finance():
    with con:
        con.execute("BEGIN")
        rejected = con.execute("SELECT count(*) FROM stg_orders_quarantine").fetchone()[0]
        if rejected:
            raise ValueError(f"publication blocked: {rejected} quarantined orders")
        candidate = {(day, currency): (count, cents) for day, currency, count, cents
                     in con.execute("SELECT * FROM mart_daily_revenue")}
        if candidate != expected:
            raise ValueError("publication blocked: source reconciliation mismatch")
        con.execute("DELETE FROM published_revenue")
        con.execute("INSERT INTO published_revenue SELECT * FROM mart_daily_revenue")

try:
    publish_finance()
except ValueError as exc:
    print(str(exc))
else:
    raise AssertionError("incomplete candidate was published")
# publication blocked: 9 quarantined orders
assert con.execute("SELECT count(*) FROM published_revenue").fetchone()[0] == 0

The 1,200 delivered rows contain 900 distinct orders. Removing 300 identical redeliveries leaves 891 usable orders and 9 quarantined orders. Core preserves all 891, including 47 without a matching customer key. A separate match flag distinguishes a missing customer from a matched customer whose country is NULL. The candidate revenue mart has six date/currency groups; revenue_cents is an integer count of that currency’s cents. Its paid/captured rule is a toy eligibility definition, not settlement or net accounting revenue, and country is current rather than historical.

The amount validator checks the complete string and bounds its length; converting the validated pieces to integers avoids binary floating-point summation. The bound and this fixture’s size keep sums within SQLite’s signed 64-bit integer range. A larger feed needs a range check or a decimal-capable engine. SQLite GLOB patterns such as [0-9]* and permissive CAST behavior are not substitutes for a format contract. Quarantine preserves the rejected order for investigation; a valid format alone does not establish a correct business amount.

Before each table is built, EXPLAIN prepares its trusted local SELECT and SQLite’s authorizer reports table reads during compilation. The check rejects a same-layer or upward dependency; an unknown table or SQL error also stops the build. Each rebuild is wrapped in an explicit transaction so a failure restores the previous derived tables in this SQLite example. The recorded core-to-mart violation is reported without building it. This local dependency check is not a sandbox for arbitrary SQL or complete cross-system lineage. See the SQLite authorizer documentation.

The candidate totals reconcile to usable synthetic orders by date and currency, and a second build produces the same result. That is not reconciliation to the complete financial population: nine amounts remain unknown. The finance gate checks quarantine and explicitly compares candidate totals to the expected totals before replacing published_revenue; on this first run no finance version is published. In a later run the same transactional failure preserves the previously approved table. The expected totals belong to this fixed input snapshot; a new source version requires a new independent reconciliation before publication. These gate checks use exceptions rather than assert, which optimized Python can disable. Orphan customers are reported separately because this particular amount total does not use country. Resolving or explicitly governing rejected orders is a prerequisite to final publication; silently dropping them is not.

Save the script in version control to make its definitions reviewable; a dictionary alone does not create version history. The database is in memory and disappears when the connection closes. The generated CSV folder remains at root; inspect it with print(root), then close con and remove that folder when finished.

4. Write three ADRs: source-evidence retention, shared revenue eligibility, and the initial processing cadence. Distinguish what the lab implements from what remains a design: delivery metadata and retention are not implemented, and daily grouping remains in the mart.

5. List what you are not building yet and the evidence that would reopen each choice. For example, consider distributed processing when measured runtime or resource limits prevent meeting the target, and faster serving when measured query latency exceeds the consumer’s requirement. Dataset size alone is not the trigger.

6. Draw stage 3. Mark the new latency path, event identities, replay destination, and reconciliation with daily results. Record any storage migration and how retained data and consumers remain supported. Keep the consumer table, requirements, diagrams, ADRs, and script together under docs/architecture/.