MapReduce

MapReduce organizes distributed processing into map tasks and, when required, reduce tasks. An input format defines logical splits; each mapper emits key/value pairs. A partitioner assigns a key’s output to a reducer.

During shuffle, reducers fetch their assigned map-output partitions and group values by key. A pair is not broadcast to every reducer. Logical splits need not equal filesystem blocks, and map-only jobs require no reduce stage.

A local combiner can shrink partial data when the aggregation is safe. Skew can concentrate work on one reducer even when there are many machines. Measure bytes and processing costs as well as record counts.

Failed or slow attempts may run again. Supported output committers coordinate output publication, but arbitrary external writes need their own retry-safe protocol. A correct final sum does not establish that an email, append, or payment happened only once.

See What Hadoop Left Behind for worked examples.

Reference: Apache documentation.