Batch and stream processing
- Lesson 31 of 40
- 15 min
- Intermediate
- 14 steps
Counting billions of events without bringing down the app's database: overnight in large batches, or continuously, as they come in.
- Results age
- 5 s
- Production DB load
- 30%
- Events per day
- 2,000M
- Exact counts
- yes
- request
- slow request
- event
- batch
- waiting in queue
Step 1 of 14
Problem
Analytics on the production database
Every play of every video is stored as a row in the production database: about 2 billion a day. A dashboard shows views per video and recomputes them every minute with a huge GROUP BY. Every figure in this lesson is an example value.
Each run scans everything so far today: by mid-morning that is already hundreds of millions of rows. The database that serves the app hits 95% load, and viewers wait while it counts.
Step 2 of 14
Solution
Batch: store now, compute later
Analytics stops querying the app's database. The app records each view as an event in cheap object storage: files, not rows. Once a night, a batch job reads the whole day, counts the views on a group of workers (4 in the drawing) and writes the results to an analytics data warehouse.
The dashboard reads from the warehouse and the production database goes back to serving only the app: its load drops to 30%. In exchange, the numbers can be up to 24 hours old.
What changed
- Event logs in object storage
- Nightly job on a group of workers
- Warehouse for analytics
- DB load: 95% → 30%
Step 3 of 14
Technique
Split the work: map, shuffle and reduce
2 billion events don't get through one machine quickly, so the work is split into phases. Map: each worker reads its share of the files and counts the views of each video. Shuffle: the counts for one video travel to the same worker. Reduce: that worker adds them up.
With twice the workers, each one reads half the data and the job takes roughly half the time. In the example, 100 workers finish in about 20 minutes.
What changed
- Workers 1 and 2: map their share
- Workers 3 and 4: reduce per video
Step 4 of 14
Problem
One viral video, one exhausted reducer
Yesterday one video went viral: it took 40% of all views, about 800 million rows. In the shuffle, all of its views go to the same reducer. That worker grinds for 3 hours while the other 99 finished in 5 minutes, and the job is not done until the slowest one is.
Splitting by key only balances the load if the keys are balanced, and real data hardly ever is. The results arrive 3 hours late: up to 27 hours old.
What changed
- One video = 40% of the data
- One reducer takes 3 h
- The job finishes 3 h late
Step 5 of 14
Solution
Combine early, split hot keys
There are two classic fixes. A combiner sums inside each mapper before the shuffle: instead of sending every "view of 7b1" row, each mapper sends a single partial count. The viral video's 800 million rows become 100 numbers.
When a key is still too heavy (in a join, for example, it can't be summarised early), salting splits it: 7b1#0 … 7b1#9 go to ten reducers and a final step adds the ten results. The job is back to about 20 minutes.
What changed
- Combiner on every mapper
- Hot key spread over 10 reducers
- Job: 3 h → 20 min
Step 6 of 14
Problem
A day behind is too late
Some questions can't wait for tonight. A video that is going viral should show up in "Trending" now. A bot farm inflating fake views has to be caught within minutes, before it gets paid for them.
Running the batch every hour helps a little, but each run reads a mountain of data again and the results still arrive late.
What changed
- Trending is a day behind
- Fake views are caught the next day
Step 7 of 14
Solution
Streaming: compute as events arrive
Now the app publishes each view to an event stream: a durable log that keeps events in the order they arrive. A stream processor reads them as soon as they come in, keeps running counts and writes the results to a fast store.
The dashboard is a few seconds behind reality instead of a day.
What changed
- Nightly batch job
- Event stream
- Stream processor
- Live counts store
- Results age: 24 h → 5 s
Step 8 of 14
Technique
Counting in time windows
A stream never ends, so "count views" needs a time frame: a window. Tumbling windows cut time into slices that don't overlap, like the views of each minute. Sliding windows do overlap: views in the last 10 minutes, updated every minute.
A sharp jump from one window to the next is exactly what a trend looks like… or fraud.
What changed
- 1-minute tumbling windows per video
Step 9 of 14
Problem
Late and duplicate events
A phone watches videos on the subway with no signal and uploads its views two hours later. Do they count for 12:04, when they happened, or for now, when they arrived? On top of that, a shaky network makes some phones send the same event twice.
Stream processors count by event time (when it happened), use a watermark to decide when a window is complete enough to publish, set a rule for stragglers, and drop duplicates by event id.
What changed
- Events 2 h late
- Duplicate events
- Watermark and dedupe by event id
Step 10 of 14
Problem
The processor falls behind
A celebrity's live broadcast triples traffic for an hour: from about 23,000 to about 70,000 events per second. The processor can't keep up and unprocessed events pile up in the stream: consumer lag reaches 4 minutes, about 17 million events.
Nothing is lost, because the stream keeps everything, but the "live" dashboard is now 4 minutes behind, and so is fraud detection.
What changed
- Traffic × 3
- Processor at 100%
- Lag: 4 min · 17M events
Step 11 of 14
Solution
Scale out with partitions
The stream is split into partitions by video id, so each video's events stay in order. Each processor instance takes some of the partitions, so adding instances adds capacity: two processors, two partitions each.
The backlog drains and lag is back to a few seconds. An autoscaler can add processors when lag grows and remove them when traffic drops.
What changed
- Second processor
- 4 partitions split 2 + 2
- Lag: 4 min → 2 s
Step 12 of 14
Technique
Counting unique viewers
Counting views is easy. Counting unique viewers is not: to do it exactly you have to remember every viewer id you have seen. With 50 million ids of 16 bytes that is about 800 MB of state, per video and per window.
A HyperLogLog sketch estimates how many distinct ids there are in about 12 KB, with a typical error under 1%. Sketches from different machines and windows also merge: the result is the same as if one machine had seen everything.
What changed
- Unique viewers per video
- Exact set → HyperLogLog sketch
- 800 MB → 12 KB per video
Step 13 of 14
Trade-off
Batch and streaming, together
Many systems run both: the stream gives fast, approximate answers, and a nightly batch recomputes the exact numbers from the full log. This is the lambda architecture.
Others keep only the stream and, to fix a bug or rebuild results, run the retained event log again through a new version of the processor. This is the kappa architecture.
What changed
- Nightly batch is back, next to the stream
Step 14 of 14
Summary
Recap
Keep heavy analytics off the production database.
Batch: process large, finished datasets cheaply; results are hours old.
Streaming: process each event as it arrives; results are seconds old, with more parts to look after.
Watch for skewed keys (combiners and salting) and consumer lag (more partitions and processors).
Handle late and duplicate events; many systems combine batch and streaming.
← → step · R replay · Space pause
You will learn
- Take analytics off the production database (covered)
- Batch processing and map/reduce (covered)
- Skewed keys: combiners and salting (covered)
- Stream processing and windows (covered)
- Late events, consumer lag and partitions (covered)
- Approximate counts, and combining batch and streaming (covered)