Key Salting
Split a hot key with a stable row-based salt, merge valid partial states, and preserve duplicates and unmatched rows when salting a join.
-
-
Data Skew
Diagnose uneven Spark work using records, bytes, and runtime metrics, and distinguish a hot key from other causes of slow tasks.
-
Python UDF
Define a Python function’s input and return contract, handle nulls, inspect its execution boundary, and compare it with equivalent built-in expressions.
-
Column Pruning
Remove unneeded columns while retaining filter, join, and grouping inputs, and distinguish a smaller projection from measured storage savings.
-
Logical and Physical Plan
Separate a query’s relational intent from execution operators, read analyzed and optimized plans, and avoid confusing plan nodes with task stages.
-
Spark DataFrame
Understand a Spark DataFrame as rows with a schema and relational plan, build expressions, and distinguish transformations from collected results.
-
Adaptive Query Execution
Read how Spark revises SQL physical plans with runtime statistics, compare initial and final plans, and avoid treating coalescing as guaranteed.
-
Spark Persistence and Lineage
Distinguish cached partitions from the recipe to recompute them, check what survives failure, and separate lineage from a durable input backup.
-
Spark Shuffle
Understand how Spark regroups records across partitions, why partial aggregation helps, and why an exchange alone does not measure network bytes.
-
Spark Job Stage and Task
Trace an action through Spark jobs, stages, and tasks, distinguish attempts from logical work, and avoid a universal shuffle-count formula.
-
Spark Driver and Executor
Separate Spark coordination from task execution, account for Python worker memory, and understand why collect puts pressure on the receiver.
-
Spark RDD
Understand an RDD as partitioned data with dependencies, distinguish transformations from actions, and keep immutability separate from source snapshots.
-
YARN
Distinguish ResourceManager, NodeManager, and ApplicationMaster roles, and understand why resource containers do not guarantee latency.
-
Combiner
Understand mapper-local combining, optional repeated execution, and why merging sums and counts is safer than averaging partial means.
-
MapReduce
Follow input splits through mapping, keyed shuffle, and reduction, and separate successful task output from repeated external effects.
-
Hive Metastore
Separate Hive Metastore metadata from table rows, check partition registration, and account for backend availability and connector behavior.
-
HDFS
Understand how HDFS separates metadata from block storage, repairs surviving replicas, and differs from a backup or a transaction layer.
-
Partition Projection
See how Athena computes partition locations from rules, why registration is ignored, and how an incorrect range can silently hide existing data.
-
Partition Pruning
Use partition values to exclude locations, distinguish pruning from row filtering, and avoid changing the answer just to reduce scanning.
-
External Table
Understand how an external table maps SQL to stored data, what registration controls, and why visibility alone is not a committed snapshot.
-
Predicate Pushdown
Understand when a reader can apply filters or skip data, why overlapping statistics are only candidates, and how to verify actual work.
-
Row Group and Column Chunk
Distinguish Parquet row groups, column chunks, and pages, and see how row group size affects writer memory, statistics-based skipping, and parallel reads.
-
Apache Arrow
Distinguish Arrow memory buffers from storage encoding, understand IPC exchange, and check the conditions needed for zero-copy access.
-
Avro
Learn how Avro uses writer and reader schemas, why nullable fields still need read defaults, and how container files differ from event framing.
-
ORC
Understand ORC stripes, column streams, and indexes, and check reader support before assuming a filter skips storage work.
-
Parquet
Understand Parquet row groups, column chunks, and optional statistics, and separate a file format from table transactions.
-
File Compaction
Combine small files while preserving records, publish replacements safely, and evaluate request savings alongside rewrite and retention costs.
-
Dataset Publication Manifest
Publish an exact immutable file set through a manifest, reject duplicate or foreign references, and retain files needed by pinned readers.
-
Conditional Object Write
Use create-only and compare-and-swap preconditions for object writes, handle conflicts, and keep publication order separate from source freshness.
-
Operational Subledger
Keep detailed operational movements traceable to source events, reconcile balances, and distinguish a signed movement table from double-entry journals.
-
Balance Roll-Forward
Explain a closing balance from its opening value and movements, align scope and cutoff, and detect errors hidden by matching aggregate totals.
-
Currency Minor Unit
Store money with its currency and scale, check provider-specific representations, and convert amounts without assuming that every currency uses cents.
-
Payment Authorization and Capture
Separate a payment hold from capture, keep expiry and partial capture rules explicit, and avoid treating either event as a bank deposit.
-
Schema Variant Column
Recognize values moved into a variant column after a type conflict, reconcile split values, and separate successful loading from correct meaning.
-
Nested Data Normalization
Flatten nested objects and arrays into tables with explicit parent links and child grain, and handle empty lists without leaving stale rows.
-
Data Connector
Evaluate a data connector by extraction coverage, state, schema handling, load behavior, and recovery rather than by a catalog entry alone.
-
Writer and Reader Schema
Use the writer schema to interpret stored data and the reader schema to resolve fields, defaults, and promotions without hiding malformed records.
-
Schema Registry and Subject
Separate a registry subject, its version history, and schema IDs; choose a compatibility scope and retain the definitions needed for replay.
-
Partition Key
Choose a partition key for ordering and load distribution, keep its encoded form stable, and test what changes when keys or partition counts change.
-
Kafka Log Compaction
Distinguish retained keyed state from full history, account for tombstone expiry, and avoid treating compaction as instant table replacement.
-
Idempotent Kafka Producer
Understand how producer identity and sequences suppress protocol retries, and why a resubmitted business event still needs its own identity.
-
In-Sync Replicas and Acknowledgements
Read Kafka acknowledgment settings together with the in-sync replica set and minimum ISR, and distinguish acceptance from external completion.
-
Committed Offset and Consumer Lag
Separate fetched position from durable progress, commit only completed prefixes, and interpret lag with retention and processing state.
-
Consumer Group and Rebalance
Share Kafka partitions among group members, resume from committed progress, and handle unfinished work when ownership changes.
-
Kafka Topic and Partition
Understand Kafka’s ordering and parallelism boundary, distinguish partition offsets, and plan key placement before changing partition counts.
-
Persistent Sending Queue
Retain outgoing telemetry across restarts while accounting for synchronization, upstream memory, retry limits, and destination acknowledgment.
-
Log Redaction
Remove sensitive log values across nested fields, failures, and output branches, and distinguish pattern checks from a privacy guarantee.
-
Multiline Log Parsing
Join stack-trace lines into the intended event, isolate stream state, and define timeout, size-limit, and malformed-record behavior.
-
Log Tailing and Rotation
Follow growing logs with file identity and byte positions, handle rotation explicitly, and separate reading progress from safe delivery.
-
Delivery Completeness
Check expected deliveries against a source calendar and deadline, including absent files, and track acceptance separately from publication.