iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Distributed data processing is the execution of a data computation across multiple networked machines. The work and often the data are divided among nodes, while software coordinates execution, communication, results, and failures. MapReduce is one model for doing this; it is not another name for all distributed data processing.
What distributed data processing means
Imagine a computation that would otherwise run on one computer. A distributed system divides the input, the computation, or both into pieces and runs those pieces on multiple connected machines. The machines can process separate portions concurrently, and the system coordinates how their work is scheduled and how the results are brought together.
Distribution is more than simply using several computers. A runtime or framework must manage work assignment, communication between nodes, result handling, and responses to failed tasks or machines. The precise design depends on the system and the kind of workload.
Recommended Free Tools
How a distributed computation works
- Divide the work: The system partitions data or computation into pieces that can be assigned to different nodes.
- Schedule execution: A runtime assigns pieces to machines and monitors their progress.
- Coordinate communication: Nodes exchange or transfer intermediate data when one part of the computation depends on another.
- Handle failures: The framework may detect failed tasks or machines and arrange for work to be retried or reassigned.
- Produce results: Partial outputs are collected or combined into the computation’s final result.
Running pieces at the same time can help with large workloads, but it does not guarantee a faster result. Coordination and data movement take time, and the runtime must manage the additional complexity of work spread across machines.
#1 Best Overall
MapReduce: one example, not the definition
MapReduce is a programming model and an associated implementation approach for processing large data sets. In the 2004 paper by Google researchers Jeffrey Dean and Sanjay Ghemawat, they described it as “a programming model and an associated implementation for processing and generating large data sets.” Read the paper.
In the model, a programmer defines two main operations:
- Map takes an input key/value pair and emits intermediate key/value pairs.
- Reduce combines values associated with the same key.
The runtime handles much of the machinery around those functions, including dividing work, scheduling it across machines, coordinating data exchange, and dealing with machine failures. MapReduce is therefore one way to build a distributed computation, not a synonym for the broader approach.
How Hadoop fits in
Apache Hadoop is a framework for applications running on a large cluster. It includes a MapReduce engine and HDFS, a distributed filesystem that stores data across machines. Hadoop divides work into fragments that can run, or be run again, on cluster nodes. Its documentation describes monitoring tasks and responding to failed nodes as part of execution.
Hadoop also uses data locality: it can schedule work on a machine where the relevant data is stored, reducing unnecessary data movement. This is useful when a job processes large portions of a dataset, but it does not eliminate all communication between machines.
When distributed processing helps—and when it may not
Distributed processing can be useful when a workload is large enough to benefit from parallel execution across machines. The potential gains depend on the work, the system, and the cost of coordinating nodes and moving data; adding machines alone does not guarantee a speedup.
Rank #4
Batch frameworks designed to scan large datasets can be a poor fit for repeated, selective queries. Apache’s Hadoop documentation notes that a database with an index will be faster than repeatedly running a MapReduce job over unindexed data for searches. See Apache Hadoop’s MapReduce tutorial.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteThe original MapReduce paper reported that Google’s implementation handled “many terabytes of data on thousands of machines.” That is a description of the authors’ implementation in the paper’s 2004 context, not a current benchmark or a general capacity guarantee. Dean and Ghemawat’s paper.
Best Value
Choosing a system for a distributed workload
Distributed processing is a broad approach, not a single product or execution strategy. When evaluating a system for a particular workload, consider:
- Whether the work is batch, streaming, interactive, or a mix.
- Which programming model and operations the application needs.
- How much data must move between nodes and whether work can run near stored data.
- Whether the priority is low latency, high throughput, or both.
- How the system recovers from failures and what consistency behavior the application requires.
- The operational complexity and cost of running the system.
These are evaluation criteria, not a ranking of current frameworks; the right choice depends on the workload and requirements.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
