Latency and throughput
- Lesson 2 of 40
- 11 min
- Beginner
- 10 steps
Two questions that sound like one: how long a request takes and how many the system can handle. You will see why they differ and what happens when the server fills up.
- p50 latency
- 120 ms
- p99 latency
- 190 ms
- Throughput
- 350 req/s
- Utilisation
- 35%
- request
- response
- slow request
- waiting in queue
- batch
Step 1 of 10
Concept
Latency and throughput are not the same
Latency: how long one request takes to go and come back. Follow one dot from the crowd to the server and back.
Throughput: how many requests the system finishes per second. Count the dots that arrive.
Step 2 of 10
Problem
Same server, seven times the people
Now 7,000 users arrive. Throughput rises to 350 req/s because the server can still cope: in this example it tops out around 500 req/s, so it is busy 70% of the time.
Look at the server's entrance: sometimes a request arrives, finds it busy and waits. Time in the queue is latency too.
What changed
- Users: 1,000 → 7,000
- Throughput: 50 → 350 req/s
- A short queue at the entrance
Step 3 of 10
Problem
Near 100%, latency explodes
A third more users and utilisation reaches 96%. Throughput barely moves (480 req/s, almost the ceiling), but latency multiplies: p99 goes from 600 ms to 4.2 s.
Requests do not arrive in a tidy line but in bursts. With headroom, a burst is served right away; without it, each burst piles onto the queue already there and takes ages to drain. That is why the curve looks like a hockey stick.
What changed
- Users: 7,000 → 9,600
- Utilisation: 70% → 96%
- Queue: 2 → 6 waiting
- p99: 600 ms → 4.2 s
Step 4 of 10
Concept
The average lies
Back to 70% utilisation. Almost every request is fast, but a few get stuck behind slow work (the dark dots). The average comes out around 180 ms and looks healthy: the few slow ones get diluted.
That is why we measure percentiles. p50 is the time half the requests stay under; p99, the time only 1 in 100 exceeds.
What changed
- Back to 7,000 users
- Some slow requests
- p99: 4.2 s → 600 ms
Step 5 of 10
Solution
More headroom, less waiting
We double the server's cores. The same 350 req/s are now only 35% of its capacity: the queue disappears and both p50 and p99 drop.
Throughput does not change because demand did not change. What we gain is headroom so bursts do not pile up.
What changed
- Server: 4 → 8 cores
- Utilisation: 70% → 35%
- The queue
- p99: 600 → 190 ms
Step 6 of 10
Concept
Distance is latency too
Now the server is in Madrid and the users in Mexico City. The server did not get any slower, yet every trip takes longer: watch how long the dots take to cross.
Light in fibre covers about 200 km per millisecond. The two cities are about 9,000 km apart, so each round trip adds at least ~90 ms, and a bit more in practice because cables do not run in a straight line.
What changed
- Server moved to Madrid
- ≥ 90 ms per round trip
- p50: 120 → 210 ms
Step 7 of 10
Technique
How many requests are in flight
One rule ties the three numbers together: in flight = throughput × latency. At 350 req/s and 0.12 s each, about 42 requests are half-done at any moment.
Each one holds a thread, a connection or some memory. If latency rises to 1 s at the same throughput, 350 are in flight, and a pool sized for 50 runs out.
What changed
- ≈ 42 requests in flight
Step 8 of 10
Problem
One search, many servers
A search asks 100 index servers at once (6 shown here) and has to wait for the slowest. Each one answers quickly 99% of the time, but with 100 of them one is almost always slow.
1 − 0.99¹⁰⁰ ≈ 63%: almost two out of three searches wait for someone's p99. What is rare for one server becomes normal for the page.
What changed
- 100 index servers (6 shown)
- The search waits for the slowest
- p99: 190 → 900 ms
Step 9 of 10
Trade-off
Batching: more throughput for a little latency
Writing analytics events one row at a time leaves the database at about 2,000 inserts per second in this example: almost all the cost is each statement's overhead.
If we collect events for up to 50 ms and write 100 rows per statement (the squares), throughput jumps to 50,000 rows/s. In exchange, each event waits up to 50 ms longer.
What changed
- Batches of 100 rows every 50 ms
- 2,000 → 50,000 rows/s
- Each event waits up to 50 ms
Step 10 of 10
Summary
Summary
Latency is the time of one request; throughput, how many finish per second.
Near 100% utilisation latency explodes: keep headroom.
Watch p99, not the average: the tail is what users feel.
Distance sets a floor no server can remove.
In flight = throughput × latency; fan-out turns the rare into the usual and batching trades latency for throughput.
← → step · R replay · Space pause
You will learn
- Tell latency and throughput apart (covered)
- Why latency explodes near 100% utilisation (covered)
- Read p50 against p99 (covered)
- Why distance sets a latency floor (covered)
- Little's law, fan-out and batching (covered)