Point-in-Time Recovery
The hard part is not replaying the log. It is knowing which second to stop at, when the bug has been quietly corrupting rows for a month.
-
-
Savepoint
A savepoint is an operator-owned snapshot for controlled restarts and planned changes. Retained checkpoints can also support some changes, subject to compatibility limits.
-
Changelog Topic
Keep state locally and back it with a log: simple to reason about, and the reason a restart takes as long as the log is long.
-
State Backend
Heap state is limited by available memory. Embedded RocksDB supports larger state within available disk capacity, with serialization and storage costs on access.
-
Stateful Stream Processing
A stateful job retains memory across events and failures. Recovery time depends on state size, the recovery path, and available resources.
-
Allowed Lateness
An engine that silently drops late records is not misbehaving. Keeping a window open to accept them costs state, which is why the default is to keep nothing.
-
Trigger and Accumulation Mode
Emitting early is easy. Deciding whether the second number replaces the first or is added to it is the part that determines whether consumers can use either.
-
Stream Windowing
An event-time window groups records by when their events occurred, rather than by when a job runs. Window shape and measure additivity determine how results can be combined.
-
Log Compaction
Time-based retention forgets by age; compaction forgets by key. One keeps a recent window of events, the other keeps the latest value for everything.
-
Topic Partition
Order exists inside a partition and nowhere else. Choosing the key decides what stays ordered; choosing the count decides how many consumers can work.
-
Spill
Spilling is not an error; it is the engine refusing to crash. It also turns the fast part of a query into the slow part, which is why it shows up in metrics.
-
Lazy Evaluation
Each step returns instantly and nothing has run. Deferring work is what lets an engine see the whole query — and why the last line gets all the blame.
-
Catalyst Optimizer
Spark’s optimizer rewrites a query in stages before anything runs. Rules do most of the work; size statistics mainly decide which join implementation you get.
-
Source Freshness Check
A build on stale input succeeds and looks fine. A freshness check compares the newest loaded row against a threshold, so the silence becomes an alert.
-
Model Materialization
The same model can be a view, a rebuilt table, or an incremental load. The choice decides build cost, query speed, and what re-running it actually does.
-
Sensor
Instead of guessing that the input has landed, a sensor waits for it. What it costs is a worker slot, some polling, and a deadline you must remember to set.
-
Data Interval
The 07:00 run usually processes yesterday, not this morning. Naming the interval a run covers is what makes a rerun mean the same thing as the original.
-
Workflow Orchestration
An orchestrator is not a fancier cron. It keeps state per run, enforces order, retries within limits — and still says nothing about whether the data is right.
-
Dead Letter Queue
Failed messages go somewhere instead of vanishing. That only helps if someone reads the queue — an unwatched DLQ looks exactly like data loss.
-
Delivery Semantics
Three promises about how many times a record arrives. The useful question is not which one a product advertises, but where its guarantee stops.
-
Replica Identity
Whether a delete event tells you only the key or the whole row is a source-side setting. It decides what history and audits downstream can reconstruct.
-
Log Sequence Number (LSN)
A log sequence number points at a place in the transaction log. It is how a pipeline knows the order of changes, where to resume, and what to ignore.
-
Incremental Snapshot
Seeding a table no longer has to mean one long locked scan. Chunks and watermarks let a snapshot run beside the change stream, and pause and resume.
-
Lambda Architecture
The Lambda architecture pairs a batch path with a fast streaming path — and then has to keep two implementations of the same logic in agreement.
-
Kappa Architecture
Kappa keeps a single streaming path. To fix logic or recompute history, it replays a retained log into a new output and switches consumers when it catches up.
-
Deterministic Ordering
Without a unique sort order, SQL may return tied rows in any order — so top-N lists, pages, and “latest record” picks can change between runs.
-
Anti-Join
An anti-join finds rows with no match on the other side — customers who never ordered. One NULL can make NOT IN return nothing; NOT EXISTS does not.
-
Late Arriving Fact
A sale recorded weeks late should still be credited to the region the customer belonged to at the time, not today’s. That is the late arriving fact problem.
-
Transaction Fact Table
One row per sale, payment, or click, at the finest grain the process records. Transaction fact tables answer “what happened” directly; state takes more work.
-
Surrogate Key
A warehouse gives each dimension row its own meaningless number, so one customer can have several historical versions, each with a key a fact can point to.
-
Late Arriving Dimension
An order arrives before the new customer’s record. Rather than drop or hold it, load it against a placeholder row and fill that row in when the data comes.
-
Durable Key
When an employee rejoins with a new number or two customer records turn out to be one person, a durable key keeps a single identity across every change.
-
Semi-Additive Fact
A balance can be summed across accounts but not across days. Semi-additive facts need a point in time or an average along the dimension where sums are nonsense.
-
Periodic Snapshot Fact Table
A periodic snapshot stores the state of every account or product once per day or month, including quiet ones, so “what was it then” becomes a simple read.
-
Non-Additive Fact
Ratios and distinct counts cannot be added up, and averaging them is usually wrong too. Store the additive parts, aggregate those, and compute the ratio last.
-
Natural Key
A customer number or SKU identifies a record in its source system. It is essential for tracing data back, and unreliable as the only key a warehouse joins on.
-
Drill-Across
To compare orders with shipments, do not join the two fact tables. Query each at the report’s grain on shared attributes, then merge the results.
-
Double Counting in Aggregation
Join an order to its shipments and the order amount appears once per shipment. The join is correct; the sum is not. This is where inflated totals come from.
-
Conformed Dimension
Orders, shipments, and returns can be compared by customer or month only if those attributes mean exactly the same thing in each. That is a conformed dimension.
-
Accumulating Snapshot Fact Table
One row per order, updated as it is picked, shipped, and delivered. An accumulating snapshot makes lead times a subtraction, but keeps one date per step.
-
Update and Deletion Anomalies
When a fact is stored in many rows, partial updates create contradictions and deleting a row can erase a fact kept nowhere else. Normalization prevents both.
-
Denormalization
Denormalization copies facts into more places so reads get simpler or faster. The price: every copy must be kept consistent by something other than the schema.
-
Conceptual, Logical, and Physical Data Models
A data model has three levels — what exists, how it is represented, and how it is stored. Each is decided by different people and changes for different reasons.
-
Mirroring (Microsoft Fabric)
Mirroring replicates an operational database into OneLake continuously, with no pipeline to build. It is still a copy, with lag and retention to manage.
-
Virtual Warehouse
A virtual warehouse is an independent Snowflake compute cluster. Size decides how fast one query runs; it is not the lever for many queries waiting at once.
-
Snowflake Credit
A Snowflake credit measures compute usage; the bill multiplies it by a price that varies by account and region. Storage and data transfer are billed otherwise.
-
Resource Monitor
A resource monitor watches warehouse credit use against a quota and can notify or suspend warehouses. It guards compute spend; it does not cap the whole bill.
-
Multi-Cluster Warehouse
A multi-cluster warehouse runs several identical clusters to reduce query queues caused by insufficient compute capacity. Each running cluster adds cost.
-
Micro-partition
Snowflake splits standard tables into small micro-partitions and records each column’s value range, so queries can skip data — if the values are grouped well.
-
Clustering Depth
Clustering depth measures how much micro-partitions overlap on a column. Lower means more data can be skipped — but query performance is still the real test.