Batch Processing & MapReduce

Overview

Batch processing runs computations over a large, bounded dataset all at once - nightly reports, log aggregation, ML training data, index builds. MapReduce is the foundational programming model for it: split the work into independent map tasks, shuffle by key, then reduce each key's group in parallel across a cluster. Modern engines like Spark generalize the idea with in-memory pipelines.

🧠 Mental model: Counting votes for a national election. Map = each local precinct tallies its own ballots in parallel. Shuffle = ship all tallies for each candidate to one place. Reduce = sum each candidate's tallies into a national total. No precinct waits on another to start counting.

Key Concepts

MapReduce has three phases. The map phase applies a function to each input record, emitting key-value pairs. The shuffle phase groups all values by key and moves them across the network so each key lands on one reducer. The reduce phase aggregates each key's values into the final output.

Because map and reduce tasks are independent, the framework provides fault tolerance for free: a failed task is simply re-run on another node from its input, since the input is immutable. This is why batch jobs recover by retrying a task rather than the whole job.

Spark improves on classic MapReduce by keeping intermediate data in memory (RDDs/DataFrames) and chaining many operations into one optimized DAG, avoiding the write-to-disk-between-every-step cost. It is often an order of magnitude faster for iterative workloads like ML.

Batch vs Stream Architectures

Architecture	Idea	Trade-off
Lambda	Run a batch layer (accurate, slow) alongside a speed/stream layer (fast, approximate); merge at query time	Accurate + fresh, but you maintain two codebases
Kappa	One stream-processing path; reprocess history by replaying the log	Simpler, single codebase; relies on a replayable log

Dimension	Batch	Stream
Input	Bounded (a dataset)	Unbounded (a continuous flow)
Latency	Minutes to hours	Milliseconds to seconds
Recovery	Re-run failed task	Checkpoint + replay offset
Best for	Reports, index/model builds, ETL	Dashboards, alerts, fraud

Trade-offs

Batch processing maximizes throughput and accuracy: it sees the whole dataset, so results are exact and easy to reason about, but they are stale by the time they land. Stream processing gives freshness at the cost of approximate, harder-to-reprocess results. The classic move is to serve fast approximate answers from a stream and correct them later with an authoritative batch recompute (the Lambda pattern), or to go Kappa and lean on log replay if a single codebase matters more than peak batch efficiency. Batch's main hazard is data skew - one hot key sends a huge group to a single reducer, which then dominates job time.

Interview Tips

Say "bounded dataset -> batch; unbounded flow -> stream," and mention they usually coexist.
Describe map -> shuffle -> reduce crisply; the shuffle is where network cost and skew live.
Name Lambda vs Kappa when asked how to get both fresh and accurate results.
Call out data skew as the top batch bottleneck and offer salting or a combiner as the fix.

Summary

Batch processing computes over a large bounded dataset for accuracy and throughput, tolerating staleness.
MapReduce splits work into parallel map tasks, shuffles by key, and reduces each group independently.
Immutable inputs make fault tolerance easy: re-run only the failed task.
Spark generalizes MapReduce with in-memory DAG pipelines, much faster for iterative jobs.
Lambda and Kappa architectures combine batch accuracy with stream freshness; watch for data skew on hot keys.