What Hadoop Left Behind

Read the system before replacing it

A Hadoop installation combines several jobs: keeping large files available across machines, allocating compute, discovering tables, and executing distributed work. Separating those jobs makes an inherited cluster easier to understand. HDFS, the Hive Metastore, and MapReduce remain useful reference points when reading storage metrics, catalog configurations, and execution plans—even when a workload uses a different engine.

The discussion does not assume Hadoop has disappeared or that moving to cloud services always costs less. Replication and data catalogs existed outside Hadoop too; the useful legacy is the concrete design and operating experience, not a claim that every modern platform descends from one stack. Start with the deployed services and workloads, because a distribution’s component list does not tell you which dependencies a particular job actually uses.

Component	Responsibility
HDFS	Files, blocks, and replica placement
Hive Metastore	Table and partition metadata
YARN	Resource allocation and application coordination
MapReduce	Map tasks, keyed shuffle, and reduce tasks

HDFS: metadata and data take different paths

The NameNode manages the filesystem namespace and file-to-block mapping. DataNodes store block replicas, report their blocks, and send heartbeats. A client asks the NameNode for locations, then transfers file data directly with DataNodes. The NameNode is not a proxy through which every data byte flows. On writes, a client uses a pipeline of DataNodes to place replicas; reads can prefer nearby replicas. Scheduling compute near data can reduce network traffic, but locality is a preference constrained by available resources.

Block size and replication are configurable. Our example uses 128 MiB blocks and three replicas spread across at least two racks. The last block can be smaller, and an empty file need not allocate a data block. Rack-aware placement reduces correlated-failure risk; replication is not a backup against accidental deletion. When a node becomes unavailable, blocks with surviving replicas can be copied elsewhere. A block with no surviving readable replica cannot be recreated just by changing metadata. Under-replicated and missing are different conditions. HDFS architecture

The namespace and file/block metadata are persisted through filesystem images and edit logs, while DataNode reports refresh replica-location knowledge. Keeping metadata in memory does not mean it exists only in memory. Monitor namespace growth, garbage collection, RPC latency, block health, disk capacity, and recovery progress together; a healthy disk total does not establish a healthy NameNode.

Availability, scale, and the small-file arithmetic

An HA deployment can pair an active NameNode with standby service, coordinate edits through JournalNodes, and use a failover protocol with fencing to prevent conflicting active writers. A Secondary NameNode performs checkpoint work; its name does not make it a hot standby. HA improves service availability but does not divide a namespace’s memory across the active and standby. Federation instead allows independent namespaces and block pools to share DataNodes; it is not automatic repartitioning of one oversized namespace. HDFS HA Federation

Small files add file entries, block metadata, RPCs, and task overhead relative to the same bytes in larger files. There is no universal 150-byte heap cost per file or block: names, attributes, implementation, and runtime configuration matter. A hypothetical 150-byte coefficient can illustrate sizing arithmetic, but does not establish deployed heap usage. Measure the deployed NameNode to size it.

Three-way replication is also not the only redundancy policy. HDFS supports erasure coding, which trades a different storage overhead for reconstruction and operational considerations. An object store manages its own durability mechanisms; do not assume its internal layout is three HDFS replicas. Storage requests, quotas, recovery, and cost still need planning when the provider manages the machines. HDFS erasure coding

Hive Metastore: a service backed by metadata

The Hive Metastore separates table discovery from file storage. A metastore service exposes metadata through an API, backed by a database containing table definitions, storage descriptors, columns, partitions, and other metadata. The database is not the table’s row store. A schema upgrade or unavailable backend can disrupt discovery even when data files remain readable. Conversely, a Spark job reading explicit file paths need not depend on that metastore. Metastore administration

For a registered partitioned table, a partition predicate can identify locations before data files are read. Filesystem contents and catalog registrations can drift. Discovery or repair is not proof of a complete write. Managed/external flags describe lifecycle behavior, but deletion and purge semantics depend on Hive version, table properties, and storage handlers. Inspect them before dropping a table. Hive DDL

Hive compatibility remains useful, but modern catalogs are not interchangeable implementations of two database tables. An Iceberg REST catalog has its own protocol for table loading and commits; authorization, credentials, namespaces, and supported table operations also vary. Match the client’s connector and protocol, not just the word catalog. Iceberg REST catalog specification

MapReduce: partitioning work by key

An InputFormat defines logical input splits; a mapper processes a split and emits key/value pairs. A split often relates to filesystem blocks, but need not equal one block: file format, compression, and split policy affect the relationship. Map-only jobs are possible. In a reduce job, a partitioner assigns each key to a reducer. Reducers fetch their partitions of map output and sort/group values by key; a pair goes to its assigned reducer, not to every reducer. Hadoop map output can spill to local disk during sorting. MapReduce tutorial

A combiner reduces data locally before the shuffle when partial results can be merged safely. It is optional and can run zero or multiple times, so correctness cannot require one combiner call. Integer sums can be combined; averaging mapper averages is wrong when mapper counts differ, while merging (sum, count) works. For example, one mapper sees [10] and another sees [20, 20, 20, 20]. Averaging their means gives 15; merging (10, 1) and (80, 4) gives 90 / 5 = 18, the mean of all five values. Floating-point sums can differ with evaluation order. For example, eight mappers each combining their own split produce forty shuffled pairs from five country totals per mapper, not five global pairs.

A hot key concentrates work on one reducer. Salting temporarily splits the key into several subkeys, followed by a second aggregation under the original key. Salt buckets are not additional reducers, and hash collisions can send multiple subkeys to one reducer. The second aggregation also costs work. In this example the combiner already makes the shuffle tiny; adding salt increases total shuffled pairs, so a lower busiest-reducer share does not establish an improvement in runtime. Record counts omit byte widths and processing cost.

InputFormat -> logical splits -> map tasks
map output -> optional local combine -> partition by key
reducers fetch assigned partitions -> sort/group -> reduce
salted aggregate -> second shuffle by original key -> final sum

Retry, commit, and intermediate state

Failed attempts can be rerun; configured speculative execution can run a second attempt for a slow task. A task can know its attempt ID, but that does not tell it whether another attempt already committed an external side effect. Output committers coordinate supported task outputs; they do not automatically deduplicate arbitrary external writes. Use a sink-side atomic idempotency key for the logical operation, including the input/run identity, or a suitable transactional protocol. An in-process set is not enough across worker crashes.

Chaining jobs through durable files repeatedly materializes intermediate results. Spark uses a DAG execution model, pipelines compatible transformations, and can retain reused data in memory or on disk through persistence. Its shuffle can still write disk data, memory pressure can cause spills, and lost state may be recomputed from lineage or recovered through a checkpoint. Spark is not simply MapReduce with every intermediate permanently in RAM. Spark RDD programming guide

A simplified materialization comparison can use equally sized 20 GB iterations. Both paths read the initial input and write the final replicated result. The retained path keeps intervening results available; the chained path rereads and rewrites each iteration. This omits shuffle, CPU, memory bandwidth, overlap, and failures. It supports a bytes comparison, not a runtime estimate.

YARN and operational dependencies

YARN separates a cluster ResourceManager, per-node NodeManagers, and an ApplicationMaster for each application. The ApplicationMaster negotiates resource containers and coordinates its tasks; a YARN container denotes a resource allocation and is not necessarily a Docker container. Queue capacities, limits, and preemption policies influence isolation. Merely creating two queues does not guarantee an interactive latency target. YARN architecture

HDFS is storage; YARN is resource management. Applications may use both, but every SQL engine need not run through YARN or every Spark job through Hive Metastore. ZooKeeper can support configured automatic failover; losing it can compromise failover without instantly stopping every running task or file read. Distinguish current service availability, future allocations, and recovery capability. ResourceManager HA

Plan capacity for the failure state you intend to survive. Twenty 12 TB nodes provide 240 TB raw, or 80 TB logical under three full replicas before other costs. Keeping 20% free on the nineteen survivors permits 60.8 TB logical under a uniform-capacity calculation. That is a ceiling under its assumptions, not a proof of rack-loss tolerance or adequate repair bandwidth. Snapshots, imbalance, disk reservations, growth, and maintenance can lower it.

Operate and migrate by workload

A distribution packages compatible component versions and management tools; it does not remove backup, identity, capacity, or recovery work. Inventory actual versions and vendor support terms rather than inferring them from the Hadoop name. Track HDFS block and namespace health, YARN queue and container failures, metastore database latency, and user-visible query success. Test metadata restoration and configured failover separately from data-node replacement.

Secure-mode Hadoop commonly uses Kerberos for authentication, with filesystem permissions, service authorization, delegation, encryption, and audit controls around it. Ranger or Sentry may appear in an inherited distribution, but deployed policy and enforcement points matter. Cloud migration changes these mechanisms; it does not reduce the whole security design to one role and policy. Hadoop secure mode

For each workload record inputs, output ownership, schedules, watermarks, SQL/UDF and library dependencies, external side effects, consumers, and rerun behavior. Join catalog inventory with file listings, scheduler history, access logs, and owners: the metastore alone is not a complete consumer inventory. Include rare month-end and recovery jobs.

Move a bounded workload using a consistent source boundary. Copying files, including with DistCp where appropriate, is one step; also translate permissions and locations, validate schema/time semantics, compare counts and aggregates, replay failures, and test the output commit behavior on the destination. Run parallel validation, define cutover and rollback, then monitor consumers before retiring the old path. Jobs with undocumented side effects need early investigation even if their cutover comes later. DistCp guide