BigQuery Architecture and Cost
BigQuery bills what a query reads or the slots it occupies. Work through column, partition, and clustering effects, dry runs, slot time, and the on-demand versus capacity choice.
-
-
Snowflake Architecture and Workload Design
Snowflake writes new files instead of editing old ones. See how that explains result reuse, the warehouse cache, Time Travel recovery, clones, and retention cost.
-
Running Data Workloads on Kubernetes
A batch job on Kubernetes can be restarted, replaced, skipped, or started twice. This article explains the Pod, Job, and CronJob rules behind that, how requests and limits decide placement and failure, and what the program must do to stay correct.
-
Containerizing Data Jobs with Docker
A worked example of a containerized batch job: a Dockerfile that rebuilds quickly, read-only inputs and separate outputs, clean shutdown, and a release record that names an image by digest.
-
Python ETL Engineering for Production
Run IDs, batch IDs, ledger commits, structured events, failure classes, retries with jitter, and signal handling, demonstrated with code that ran.
-
Designing Maintainable Data Workflows
How to structure a data job so its inputs are visible, its effects are local, and changes can be checked: dependency injection, contract tests, configuration precedence and validation, and effective-configuration records.
-
Asset-Oriented Orchestration: Dagster and Prefect
Declare data assets instead of tasks, record what each production run created, map partitions to find what a correction affects, and gate publication with checks.
-
Production Airflow: Idempotency and Backfills
Reproduce the failures that corrupt data in scheduled pipelines, from duplicate writes after a lost acknowledgement to a backfill that overwrites a correction, and fix each one.
-
Curating Training Data and Managing Feedback
A training set is a product with a specification. This post covers sampling, labels, provenance, dataset versions, feedback quality, contamination checks, and privacy limits.
-
Preserving Access Controls in Retrieval Data
A retrieval index is a copy of documents, and copies do not inherit permissions. This post follows permissions from source ACLs through identity, filtering, revocation, and leakage tests.
-
Document Pipelines for Retrieval
A retrieval index is derived data. Track which document version produced each chunk, refresh only what changed, delete everywhere, and validate before publishing.
-
Feature Pipelines and Point-in-Time Correctness
A feature value is a claim about what a system knew at prediction time. Learn the clocks, joins, parity checks, and backfill rules that keep that claim true.
-
Data Platform Disaster Recovery and Cost
A replica copies corruption and a restore needs more than files. Learn restore dependencies, corruption recovery, drills, billing units, and how to weigh reliability against cost.
-
Data SLIs, SLOs, and Incident Response
A green job does not mean a reliable dataset. Learn to define data indicators, set objectives, alert on budget burn, find silent failures, and tell consumers what to do.
-
Access, Retention, and Erasure Across Data Copies
A deleted row is not an erased person. Follow classification, access rules, retention clocks, derived copies, backups, and audit evidence across a data platform.
-
Master Data and Entity Resolution Pipelines
Follow records from several systems through blocking, scoring, clustering, and survivorship, and see how thresholds, chaining, and corrections decide merge quality.
-
Metadata and Lineage for Change Impact Analysis
See how run metadata, column-level lineage, coverage measurement, and consumer discovery turn “who uses this column?” into a candidate list you can validate.
-
Data Quality Invariants and Reconciliation
Learn what exact invariants, statistical checks, and reconciliation against an independent reference can and cannot detect, with detection and false-positive arithmetic.
-
Operating Streams: Backpressure and Replay
Lag, replay, and upgrades are where stateful streaming jobs usually fail. Work through the arithmetic and the procedures that keep restarts and rollouts safe.
-
Stateful Stream Processing and Checkpoint Recovery
A checkpoint restores a job’s state, not the outside world. Trace restart, sink commits, and crash points to see where exactly-once ends.
-
Event Time, Watermarks, and Late Data
A watermark is an estimate, not a fact. See how the delay you choose trades result latency against dropped or revised records, and what it costs in state.
-
Semantic Layers and Consistent Metric Definitions
One question, “what was March revenue?”, can honestly return 600, 550, or 100 from the same seven orders. This post shows how metric contracts, aggregation rules, time-window rules, versions, and consumer tests make one answer.
-
Dimensional Modeling and Historical Restatement
A customer moves region, a sale arrives late, and a correction shows the move happened earlier. This post shows how a dimensional model attributes facts, restates history on purpose, and proves the totals still agree.
-
Warehouse Workload Isolation and Concurrency
When dashboards wait behind an export, resizing may not help. Learn how compute separation, queues, workload classes, and access controls shape warehouse performance.
-
Partitioning, Compaction, and Table Maintenance
Lakehouse tables degrade through small files, drifting layouts, and old snapshots. See how sizing, pruning, compaction, and expiry work and how to validate their cost.
-
Open Table Formats: Snapshots and Catalogs
A table format decides which files are the table, commits changes atomically, and lets readers keep a consistent snapshot. See how the mechanisms and their limits work.
-
Data Pipeline CI/CD and Environment Promotion
A code rollback does not undo data a bad release already wrote. See how fixtures, contract tests, isolated builds, and staged promotion limit that damage, and how to repair it.
-
Airflow Operations: Retries and Backfills
Follow one failed run through logical time, retries, catchup, backfill, worker and scheduler failure, and see which rules keep a repeated run from corrupting data.
-
Transformation as Code: Models and Tests
Trace how a SELECT statement becomes a model in a dependency graph, how data tests and unit tests differ, and why an incremental model can drift from a full rebuild.
-
Apache Airflow Fundamentals
Follow an Airflow workflow from definition to execution, choose interval and retry rules, and understand what a successful run actually confirms.
-
Single-Node Analytics: DuckDB, Polars, and Arrow
Compare DuckDB, Polars, and Arrow with Parquet queries, null-safe joins, and interchange checks. Separate result correctness from memory and speed claims.
-
Testing and Debugging Spark Data Pipelines
Build fixtures and independent expected results, test real storage boundaries, and reduce Spark failures into useful regression tests.
-
Building Incremental and Idempotent Pipelines
Define incremental bounds and retry identity, preserve newer updates and deletes, and coordinate publication, progress, and backfills.
-
Optimizing Spark Jobs in Production
Diagnose production Spark work, preserve query results, compare repeated runs, and decide whether a faster job also meets cost and delivery requirements.
-
Spark Partitioning, Shuffles, and Data Skew
Understand partition distributions, salted aggregates and joins, broadcast plans, coalesce, and Parquet file counts while preserving query semantics.
-
Spark DataFrames and Spark SQL
Understand Spark transformations through explicit schemas, join rules, numeric limits, reporting days, and Python UDF versus built-in expression semantics.
-
Apache Spark Architecture and Execution
Understand Spark drivers, executors, partitions, memory limits, caching, and the conditions for recomputation and recovery.
-
What Hadoop Left Behind
Read inherited Hadoop systems through storage, catalogs, replica repair, mapper-local combining, retry boundaries, and workload migration.
-
External Tables and the Lake/Warehouse Boundary
An external table exposes files to SQL. Learn when to load a managed copy and how registration, updates, and snapshots affect correctness.
-
Data Formats: Parquet, ORC, Avro, and Arrow
Learn what each format stores, how readers skip data, and which schema and layout choices change analytical work.
-
Storage Tiering, Lifecycle, and Archival
Choose storage tiers from access and recovery needs, lifecycle filters, version retention, holds, and restoration checks.
-
Designing an Object-Storage Data Lake
Build a data lake with discoverable datasets, immutable runs, conditional publication, safe cleanup, and layouts matched to queries.
-
Integrating Payment, Commerce, and Finance Data
Trace payment movements across orders, wallets, processor balances, and finance reports while preserving currency, timing, and correction history.
-
Integrating Marketing and Attribution Data
Build reproducible marketing reports with campaign mapping, conversion paths, attribution rules, source revisions, and reconciliation.
-
Managed Ingestion: Fivetran, Airbyte, and dlt
Decide when to buy, operate, or write a connector, and verify its cursor, nested tables, schema changes, failure handling, and total cost.
-
Designing Topics and Event Contracts
When one team quietly makes a change that breaks the contract it sends under, every team that depends on the changed part can break at once. A contract makes the change visible before it ships.
-
Kafka Fundamentals
Understand Kafka ordering, delivery guarantees, replay, commit boundaries, retention gaps, and hot partitions.
-
Log and Telemetry Collection
Learn how collectors follow rotating logs, handle outages, redact sensitive fields, and deliver records with explicit retry and deduplication rules.
-
Ingesting Data from APIs, Files, and Storage
Collect partner data with explicit identity, revision, retry, and completeness rules.
-
Database Ingestion and Change Data Capture
Build database copies with explicit snapshot boundaries, safe progress tracking, delete handling, and reconciliation.