Cloud Services Layer
The cloud services layer coordinates Snowflake — sign-in, metadata, query planning. It is billed only above 10% of daily warehouse use.
-
-
Secure Data Sharing
Secure Data Sharing grants other Snowflake accounts read-only access to live data without copying it — until the consumer is in another region.
-
Deletion Vector
A deletion vector is a bitmap that marks which rows of an immutable data file are deleted — cheap to write, a small tax on reads until the file is rewritten.
-
Type Promotion
Type promotion widens a column’s type — int to long, float to double — without rewriting old files. See which widenings are allowed and why narrowing never is.
-
Field ID
A field ID gives each column a permanent number, so a renamed or reordered column still lines up with old data files, and a new column never inherits old data.
-
Table Format
A table format turns a pile of immutable files into a table: which files count right now, what the schema is, and how a change becomes visible all at once.
-
Massively Parallel Processing (MPP)
MPP splits one query across many nodes working on different parts of the data at once — fast until the data has to move between them or divides unevenly.
-
Vectorized Execution
Vectorized execution moves batches of values between query operators instead of single rows — spreading call overhead thin and keeping the CPU’s work in cache.
-
Columnar Storage
Columnar storage keeps each column’s values together so a query reads only the columns it names — at the price of expensive row lookups and small writes.
-
Schema-on-Read
Schema-on-read stores data as it arrives and decides its structure at query time. See what that flexibility buys, what it pushes downstream, and when to use it.
-
JSON Schema
JSON Schema describes the structure a JSON document must have. A sentiment-result example shows why a document can conform while its meaning is wrong.
-
Accepted Break
Not every difference found in a data migration is worth fixing. An accepted break records the ones the team will live with: who approved them, why, how far they extend, and when the approval lapses.
-
Dual Running
During a data platform migration, both pipelines process the same inputs and a comparison reports what differs. What makes the comparison trustworthy is the shared input, the key used to match rows, and a window long enough to see rare events.
-
Dry Run
Learn what a dry run checks before a command takes effect, how BigQuery uses it to estimate query bytes, and why an estimate is not a limit.
-
Clustered Table
Learn how clustering sorts rows inside a table, why column order decides which queries benefit, and what it costs in estimate accuracy.
-
BigQuery Slot
Learn what a BigQuery slot is, how baseline and autoscale slots differ, and why idle slots move between reservations but never across editions.
-
Time Travel
Learn how a warehouse keeps prior versions queryable for a retention window, what that recovers, and why longer retention shows up as storage cost.
-
Result Cache
Learn how a stored query result is reused, the exact conditions that allow it, and why timing a query twice measures the cache instead of the query.
-
Measure Additivity
Learn which measures can be summed across every dimension, which only across some, and why a ratio stored per row generally breaks when it is rolled up.
-
Resource Requests and Limits
Distinguish resource requests used for placement from runtime limits, interpret CPU and memory differently, and measure demand before setting values.
-
Kubernetes Pod
Understand a Pod as a scheduling unit, distinguish container restart from Pod replacement, and keep durable output separate from an attempt’s identity.
-
Kubernetes Job and CronJob
Separate finite work from scheduled Job creation, scope concurrency and deadline policies, and keep repeat-safe output independent of controller success.
-
Image Tag and Digest
Separate a movable image tag from a content digest, retain platform identity, and update pinned artifacts through a reviewed build and test process.
-
Container Image and Container
Distinguish a packaged image from a running container, identify the writable layer, and account for the host kernel, architecture, and external inputs.
-
Bind Mount and Volume
Choose host-path sharing or Docker-managed storage, check daemon-side paths and permissions, and keep persistence separate from backups and publication.
-
Structured Logging
Emit stable event fields for data jobs, separate candidate counts from committed results, and keep JSON formatting distinct from redaction and delivery.
-
Dependency Injection
Pass a workflow its reader or publisher from outside, test the dependency contract, and keep fake-adapter evidence separate from real integration behavior.
-
Configuration Validation
Check configuration keys, types, ranges, and relationships before work begins, and distinguish a valid setting from a safe destination or correct rule.
-
Asset Partition Mapping
Map a downstream asset partition to the upstream partitions it needs, including windows, and keep logical dependencies separate from storage layout.
-
Asset Materialization
Connect an asset’s production event to its physical output, version, and partition, without confusing execution history with durable or complete data.
-
Asset Check
Attach a named test to a data asset, configure whether failure blocks downstream work, and distinguish blocking from rollback or source completeness.
-
Deferrable Operator and Sensor
Release workers during supported waiting, distinguish deferral from reschedule mode, and account for triggerer capacity and pool policy.
-
Compare-and-Swap
Check the expected version before publishing a replacement, handle conflicts deliberately, and distinguish CAS from idempotency and fencing.
-
Airflow Pool
Limit participating Airflow tasks with weighted pool slots, distinguish concurrency from request rate, and plan how deferred tasks count.
-
Airflow Timetable
Learn how Airflow timetables define run timing and data intervals, compare trigger and interval schedules, and check time boundaries before deployment.
-
Airflow TaskFlow API
Separate TaskFlow graph construction from task execution, follow returned values through XCom, and recognize the limits of ordinary Python-looking calls.
-
Airflow Catchup
Understand how Airflow catchup selects missing intervals, compare it with retries and backfills, and avoid confusing scheduled work with complete data.
-
Trigger Rule
Choose which upstream states allow a task to run, distinguish all_success from all_done, and check how leaf tasks affect the overall run status.
-
Airflow XCom
Pass small values or versioned data references between Airflow tasks, and separate metadata exchange from durable shared storage.
-
Data Interval and Logical Date
Use a run’s intended data interval instead of wall-clock time, check boundaries, and account for timetable-specific logical dates.
-
Task Instance and Retry
Track an Airflow task within a run, distinguish attempts from executions, and make repeated destination effects safe before adding retries.
-
Airflow DAG and DAG Run
Separate an Airflow workflow definition from each execution, trace task dependencies, and understand why different runs can overlap.
-
Out-of-Core Processing
Data bigger than RAM does not require a cluster. Engines stream what they can and spill the rest to disk — with limits worth knowing before you rely on it.
-
Polars
Build Polars expressions over DataFrames, scan files lazily, and check streaming support, result size, and null behavior before relying on a plan.
-
DuckDB
Use DuckDB for analytical SQL over files and tables, plan memory and temporary storage, and keep local results separate from production guarantees.
-
Incremental Processing
Update results from bounded changes, identify affected outputs, and distinguish incremental work from complete capture and safe retries.
-
Dynamic Allocation
Adjust Spark executor allocation with demand, preserve shuffle access, and separate executor release from immediate capacity or lower resource charges.
-
Spark Event Log
Retain Spark execution events for later diagnosis, configure shared history access, and distinguish event evidence from application logs and recovery data.
-
Repartition and Coalesce
Choose redistribution or narrow partition merging, preserve the needed downstream layout, and distinguish task counts from committed data files.
-
Broadcast Join
Understand the build-side copy in a broadcast join, preserve match multiplicity, and avoid treating file size as a safe executor memory budget.