Batch and stream processing

  • Lesson 31 of 40
  • 15 min
  • Intermediate
  • 14 steps

Counting billions of events without bringing down the app's database: overnight in large batches, or continuously, as they come in.

Results age
5 s
Production DB load
30%
Events per day
2,000M
Exact counts
yes
viewersVideo appVideo appproduction DBDashboardDashboardlogsWorker 1Worker 2Worker 3Worker 4warehouseEventstreamEvent streamProcessorProcessorProcessor2Processor 2live counts95%GROUP BY video · every minute30%nightly jobappend eventsread 1 daywrite results30%mapmapreducereduceCounting 2 billion viewsmapcounts its shareshufflesame video → same workerreducesums per video100 workers≈ 20 minexample values30%40% of the data · 3 h5 min30%map + combinemap + combinereduce · evenreduce · evenTaming a hot keyviral video800M rows → one reducercombinereach mapper sends 1 countsalting7b1#0 … 7b1#9 → 10 reducersjob3 h → 20 min30%trending: yesterday’snightly job30%trending: nowpublish each viewupdate counts30%trending: now1-min windowsViews of 7b1 per minute12:019012:0211012:0365012:043,200 · trendingexample values30%watermark: 12:05drops duplicatesevent from 2 h agoEvent time vs arrival timeview at 12:04arrives at 14:10watermarkallows 5 min of delay2 h lateits window is fixed laterrepeated iddroppedexample times30%100%lag: 4 min · 17M eventslive: 4 min behind30%partitions 0–1partitions 2–34 partitions · lag 2 strending: now30%unique ≈ 49.7MHyperLogLog per videoUnique viewers of 7b1exactkeep every id seen50M ids × 16 B≈ 800 MBHyperLogLog12 KB · error < 1%mergesketches combineexample values30%nightly · exactlive · approximateBatch vs streamingbatchstreamingfreshnesshourssecondscomplexitylowhigherreruneasyreplay the log30%nightly · exactlive · approximate
  • request
  • slow request
  • event
  • batch
  • waiting in queue

Step 1 of 14

Problem

Analytics on the production database

Every play of every video is stored as a row in the production database: about 2 billion a day. A dashboard shows views per video and recomputes them every minute with a huge GROUP BY. Every figure in this lesson is an example value.

Each run scans everything so far today: by mid-morning that is already hundreds of millions of rows. The database that serves the app hits 95% load, and viewers wait while it counts.

Step 2 of 14

Solution

Batch: store now, compute later

Analytics stops querying the app's database. The app records each view as an event in cheap object storage: files, not rows. Once a night, a batch job reads the whole day, counts the views on a group of workers (4 in the drawing) and writes the results to an analytics data warehouse.

The dashboard reads from the warehouse and the production database goes back to serving only the app: its load drops to 30%. In exchange, the numbers can be up to 24 hours old.

What changed

  • Event logs in object storage
  • Nightly job on a group of workers
  • Warehouse for analytics
  • DB load: 95% → 30%

Step 3 of 14

Technique

Split the work: map, shuffle and reduce

2 billion events don't get through one machine quickly, so the work is split into phases. Map: each worker reads its share of the files and counts the views of each video. Shuffle: the counts for one video travel to the same worker. Reduce: that worker adds them up.

With twice the workers, each one reads half the data and the job takes roughly half the time. In the example, 100 workers finish in about 20 minutes.

What changed

  • Workers 1 and 2: map their share
  • Workers 3 and 4: reduce per video

Step 4 of 14

Problem

One viral video, one exhausted reducer

Yesterday one video went viral: it took 40% of all views, about 800 million rows. In the shuffle, all of its views go to the same reducer. That worker grinds for 3 hours while the other 99 finished in 5 minutes, and the job is not done until the slowest one is.

Splitting by key only balances the load if the keys are balanced, and real data hardly ever is. The results arrive 3 hours late: up to 27 hours old.

What changed

  • One video = 40% of the data
  • One reducer takes 3 h
  • The job finishes 3 h late

Step 5 of 14

Solution

Combine early, split hot keys

There are two classic fixes. A combiner sums inside each mapper before the shuffle: instead of sending every "view of 7b1" row, each mapper sends a single partial count. The viral video's 800 million rows become 100 numbers.

When a key is still too heavy (in a join, for example, it can't be summarised early), salting splits it: 7b1#0 … 7b1#9 go to ten reducers and a final step adds the ten results. The job is back to about 20 minutes.

What changed

  • Combiner on every mapper
  • Hot key spread over 10 reducers
  • Job: 3 h → 20 min

Step 6 of 14

Problem

A day behind is too late

Some questions can't wait for tonight. A video that is going viral should show up in "Trending" now. A bot farm inflating fake views has to be caught within minutes, before it gets paid for them.

Running the batch every hour helps a little, but each run reads a mountain of data again and the results still arrive late.

What changed

  • Trending is a day behind
  • Fake views are caught the next day

Step 7 of 14

Solution

Streaming: compute as events arrive

Now the app publishes each view to an event stream: a durable log that keeps events in the order they arrive. A stream processor reads them as soon as they come in, keeps running counts and writes the results to a fast store.

The dashboard is a few seconds behind reality instead of a day.

What changed

  • Nightly batch job
  • Event stream
  • Stream processor
  • Live counts store
  • Results age: 24 h → 5 s

Step 8 of 14

Technique

Counting in time windows

A stream never ends, so "count views" needs a time frame: a window. Tumbling windows cut time into slices that don't overlap, like the views of each minute. Sliding windows do overlap: views in the last 10 minutes, updated every minute.

A sharp jump from one window to the next is exactly what a trend looks like… or fraud.

What changed

  • 1-minute tumbling windows per video

Step 9 of 14

Problem

Late and duplicate events

A phone watches videos on the subway with no signal and uploads its views two hours later. Do they count for 12:04, when they happened, or for now, when they arrived? On top of that, a shaky network makes some phones send the same event twice.

Stream processors count by event time (when it happened), use a watermark to decide when a window is complete enough to publish, set a rule for stragglers, and drop duplicates by event id.

What changed

  • Events 2 h late
  • Duplicate events
  • Watermark and dedupe by event id

Step 10 of 14

Problem

The processor falls behind

A celebrity's live broadcast triples traffic for an hour: from about 23,000 to about 70,000 events per second. The processor can't keep up and unprocessed events pile up in the stream: consumer lag reaches 4 minutes, about 17 million events.

Nothing is lost, because the stream keeps everything, but the "live" dashboard is now 4 minutes behind, and so is fraud detection.

What changed

  • Traffic × 3
  • Processor at 100%
  • Lag: 4 min · 17M events

Step 11 of 14

Solution

Scale out with partitions

The stream is split into partitions by video id, so each video's events stay in order. Each processor instance takes some of the partitions, so adding instances adds capacity: two processors, two partitions each.

The backlog drains and lag is back to a few seconds. An autoscaler can add processors when lag grows and remove them when traffic drops.

What changed

  • Second processor
  • 4 partitions split 2 + 2
  • Lag: 4 min → 2 s

Step 12 of 14

Technique

Counting unique viewers

Counting views is easy. Counting unique viewers is not: to do it exactly you have to remember every viewer id you have seen. With 50 million ids of 16 bytes that is about 800 MB of state, per video and per window.

A HyperLogLog sketch estimates how many distinct ids there are in about 12 KB, with a typical error under 1%. Sketches from different machines and windows also merge: the result is the same as if one machine had seen everything.

What changed

  • Unique viewers per video
  • Exact set → HyperLogLog sketch
  • 800 MB → 12 KB per video

Step 13 of 14

Trade-off

Batch and streaming, together

Many systems run both: the stream gives fast, approximate answers, and a nightly batch recomputes the exact numbers from the full log. This is the lambda architecture.

Others keep only the stream and, to fix a bug or rebuild results, run the retained event log again through a new version of the processor. This is the kappa architecture.

What changed

  • Nightly batch is back, next to the stream

Step 14 of 14

Summary

Recap

Keep heavy analytics off the production database.

Batch: process large, finished datasets cheaply; results are hours old.

Streaming: process each event as it arrives; results are seconds old, with more parts to look after.

Watch for skewed keys (combiners and salting) and consumer lag (more partitions and processors).

Handle late and duplicate events; many systems combine batch and streaming.

You will learn

  • Take analytics off the production database
  • Batch processing and map/reduce
  • Skewed keys: combiners and salting
  • Stream processing and windows
  • Late events, consumer lag and partitions
  • Approximate counts, and combining batch and streaming