Building Incremental and Idempotent Pipelines
Define incremental bounds and retry identity, preserve newer updates and deletes, and coordinate publication, progress, and backfills.
-
-
Optimizing Spark Jobs in Production
Diagnose production Spark work, preserve query results, compare repeated runs, and decide whether a faster job also meets cost and delivery requirements.
-
Spark Partitioning, Shuffles, and Data Skew
Understand partition distributions, salted aggregates and joins, broadcast plans, coalesce, and Parquet file counts while preserving query semantics.
-
Spark DataFrames and Spark SQL
Understand Spark transformations through explicit schemas, join rules, numeric limits, reporting days, and Python UDF versus built-in expression semantics.
-
Apache Spark Architecture and Execution
Understand Spark drivers, executors, partitions, memory limits, caching, and the conditions for recomputation and recovery.
-
What Hadoop Left Behind
Read inherited Hadoop systems through storage, catalogs, replica repair, mapper-local combining, retry boundaries, and workload migration.
-
External Tables and the Lake/Warehouse Boundary
An external table exposes files to SQL. Learn when to load a managed copy and how registration, updates, and snapshots affect correctness.
-
Data Formats: Parquet, ORC, Avro, and Arrow
Learn what each format stores, how readers skip data, and which schema and layout choices change analytical work.
-
Storage Tiering, Lifecycle, and Archival
Choose storage tiers from access and recovery needs, lifecycle filters, version retention, holds, and restoration checks.
-
Designing an Object-Storage Data Lake
Build a data lake with discoverable datasets, immutable runs, conditional publication, safe cleanup, and layouts matched to queries.
-
Integrating Payment, Commerce, and Finance Data
Trace payment movements across orders, wallets, processor balances, and finance reports while preserving currency, timing, and correction history.
-
Integrating Marketing and Attribution Data
Build reproducible marketing reports with campaign mapping, conversion paths, attribution rules, source revisions, and reconciliation.
-
Managed Ingestion: Fivetran, Airbyte, and dlt
Decide when to buy, operate, or write a connector, and verify its cursor, nested tables, schema changes, failure handling, and total cost.
-
Designing Topics and Event Contracts
When one team quietly makes a change that breaks the contract it sends under, every team that depends on the changed part can break at once. A contract makes the change visible before it ships.
-
Kafka Fundamentals
Understand Kafka ordering, delivery guarantees, replay, commit boundaries, retention gaps, and hot partitions.
-
Log and Telemetry Collection
Learn how collectors follow rotating logs, handle outages, redact sensitive fields, and deliver records with explicit retry and deduplication rules.
-
Ingesting Data from APIs, Files, and Storage
Collect partner data with explicit identity, revision, retry, and completeness rules.
-
Database Ingestion and Change Data Capture
Build database copies with explicit snapshot boundaries, safe progress tracking, delete handling, and reconciliation.
-
Designing Event Data for Product Analytics
Design event contracts that keep product metrics interpretable as identifiers, client clocks, delivery retries, and application versions change.
-
ETL, ELT, and Data-Pipeline Design Patterns
ETL and ELT differ in where the transformation runs. The patterns underneath them decide whether a pipeline can be rerun without breaking anything.
-
Batch, Micro-Batch, and Streaming
Choose processing modes from freshness needs, then test how late events, duplicates, and recovery affect the results.
-
Data Modeling for OLTP and OLAP Systems
How row meaning, identity, dependencies, aggregation, and time shape operational and analytical data models.
-
Designing End-to-End Data Architecture
How consumer requirements determine data contracts, logical layers, physical deployment, consistency boundaries, and recovery decisions.
-
Researching Errors and Organizing Analytical Work
A ride from 23:00 to 06:00 reports minus seventeen hours. Follow the evidence from a failing example to a tested rule, and record where that rule applies.
-
Software Engineering Habits for Data Work
Make data changes easier to verify and recover with tests, focused reviews, incremental refactoring, and useful operational documents.
-
Relational Database Internals
Explore how indexes, statistics, WAL, snapshots, and locks shape query behavior through SQLite examples and PostgreSQL concurrency mechanisms.
-
CTEs and Temporary Tables
A WITH clause is not a temporary table. Compare scope, lifetime, visibility, and staleness with DuckDB examples.
-
SQL for Data Engineers
Understand SQL through examples for filtering, cleaning, joins, aggregation, windows, and transactions in DuckDB.
-
Pandas Foundations
A spreadsheet of support tickets, five questions, one table. Filter it, group it, join it to a lookup table, and see where pandas quietly changes the answer.
-
Python for Data Engineers
Build installable Python data jobs with explicit configuration, input validation, tests, and a deliberate policy for rejected records and reruns.
-
Python Functions, Modules, and Classes
Copy-pasted cleaning logic breaks when the rule changes. Wrap it in a function, document it, import shared code, and let a class carry data with its checks.
-
Python Collections and Comprehensions
One name per value stops working at scale. Pick the container that matches the question you ask of the data, then build new collections with comprehensions.
-
Python Strings, Conditions, and Loops
Text rows rarely arrive tidy. Slice and strip them, decide with conditions, repeat the work with loops, and format the result for a reader.
-
Python Variables, Types, and Notebooks
A first Python session in a notebook: what a name refers to, why a value’s type decides what + does, and how the order of cell runs changes results.
-
Linux, the Shell, and Git for Data Work
Inspect files, permissions, processes, and logs with Bash, then track and review changes with Git.
-
Network Retries and Timeouts for Data Pipelines
Trace network and storage delays, handle uncertain writes safely, and verify a download before making it visible.
-
How Computers Store Data
Before any pipeline runs, one agreement has to hold: what do these bytes mean?
-
What Data Engineering Is
How data engineering connects source meaning, pipeline state, platform architecture, and consumer commitments.
-
Measuring an AI Development Tool Rollout
Dashboards can credit an AI coding tool for gaps that predate it. This guide covers pilot design, paired speed and quality measures, review cost, adoption, power, and gates.
-
Coding Agent Permissions and Sandboxing
A coding agent turns model output into actions. This guide shows how isolation, scoped credentials, argument-level tool rules, audit, and limits bound what a steered agent can do.
-
Testing and Reviewing AI-Generated Code in CI
Generated code can pass its own tests and still be wrong. Build CI checks that stay independent of the generator, and define what a human must approve.
-
Integrating AI Coding Tools with Repositories
Before an AI coding tool touches enterprise code, decide what it can read, what leaves the machine, how long copies persist, and how you would audit it afterward.
-
Platform Unit Economics and Cost Controls
A platform bill is one number. Learn to allocate it, divide it by a workload unit that drives cost, price headroom and commitments, and tune without breaking objectives.
-
Incident Response and Recovery Testing
Recovery is a claim until it has been exercised. This article covers incident roles, runbooks, containment, RTO and RPO, restore tests, and analysis.
-
Capacity, Load Shedding, and Resilience Testing
Services fail slowly as queues grow, not when demand first exceeds capacity. This article covers queues, capacity, overload controls, fault injection, and recovery.
-
SLOs, Alerting, and Error Budgets
Pages should mean users are hurt faster than the objective allows. This article builds SLIs, burn-rate alerts, and an error budget policy.
-
Metrics, Logs, and Traces for Services
Telemetry only helps if it carries context, stays affordable, and respects privacy. This article instruments a service with OpenTelemetry and examines cardinality, sampling, propagation, and redaction choices.
-
Retiring Platform Capabilities
Retirement fails when a rare consumer surfaces after the switch-off. This article covers dependency discovery, migration support, compatibility windows, communications, and the evidence to collect before removal.
-
Measuring Platform Adoption and Developer Friction
Adoption counts say who touched a platform, not whether work got easier. This article builds workflow measures, qualitative research, and baselines, and states what they cannot prove.
-
Designing Platform APIs and Extension Boundaries
Portals, pipelines, and extensions all call the platform API. This article covers its lifecycle, tenancy, idempotency, extension permissions, and how to change it compatibly.