Vertical or horizontal

  • Lesson 4 of 40
  • 11 min
  • Beginner
  • 11 steps

Your app lives on a single server and traffic keeps growing. There are two ways to get more capacity, and each one has its price.

Users
500,000
Requests
20,000 req/s
p95 latency
95 ms
Errors
0%
Cost $/month
14,400
500,000 usersApp serverApp serverServer AServer AServer BServer BServer CServer CServer DServer DServer EServer EServer FServer F1%4 vCPU · 16 GB100%480 req/s arrive, 400 fit30%16 vCPU · 64 GB100%192 vCPU · 768 GB: no biggerdown54%52%52%50%Capacity (example)1 × 192 vCPU≈ 19,200 req/s4 × 96 vCPU≈ 38,400 req/sif one fails≈ 28,800 req/sthe load is 20,000 req/s5%5%6%5%10%11%Autoscaling ruletargetaverage CPU ≈ 50%minimum2 serversmaximum8 serversaverage cost$10,800 a month12 h with 4 servers + 12 h with 2100%100%booting · ≈ 3 min52%52%52%down: 1 in 4 users without serviceUp versus outhowbigger boxmore boxescodeunchangedstatelessceilingthe hardwarealmost noneon failureall downa slice downto growmigrateadd onecostless per $linear
  • request
  • response
  • waiting in queue

Step 1 of 11

Concept

One machine does everything

At first your app runs on a single server: every time someone opens a screen or saves something, the request leaves their device, reaches that machine and comes back with the response.

With 100 users online there are about 4 requests per second. The machine has 4 vCPUs and handles about 400 req/s, so it runs at 1%.

Step 2 of 11

Problem

A video goes viral

Overnight you have 120 times more users: 12,000 online and 480 req/s. The server only handles about 400, so 80 per second are too many.

CPU is stuck at 100%, requests wait in a queue, p95 latency climbs to 2.4 s and about 17% (80 out of every 480) end in a timeout.

What changed

  • Users 100 → 12,000
  • CPU 1% → 100%
  • Timeouts: ≈ 17%

Step 3 of 11

Solution

Path 1: a bigger machine

The most direct way out is moving to a server with more vCPUs, more memory and faster disks. Going from 4 to 16 vCPUs gives about 1,600 req/s of capacity.

The code does not change at all. This is called vertical scaling (scaling up). At 480 req/s CPU drops to 30%.

What changed

  • Server 4 → 16 vCPUs
  • CPU 100% → 30%
  • Cost $150 → $600 a month

Step 4 of 11

Trade-off

Until there is no bigger machine

The product keeps growing to 500,000 users and 20,000 req/s. You move up a size again and again until the biggest machine in the catalogue: 192 vCPUs. On paper that is 19,200 req/s, but in practice it delivers about 17,600.

Software does not use 192 cores as well as 16: locks, shared memory and a single disk become bottlenecks. 12% of requests fail. And every size jump is a migration with a restart.

What changed

  • Users 12,000 → 500,000
  • Biggest machine reached
  • Cost $600 → $7,200 a month

Step 5 of 11

Problem

Everything hangs on one box

There is a worse problem than the ceiling: if that machine goes down, everything goes down. A buggy deploy, a full disk or a dead power supply is enough.

A component whose failure takes the whole system down is a single point of failure (SPOF).

What changed

  • App server down
  • Errors 12% → 100%

Step 6 of 11

Solution

Path 2: more identical machines

Instead of one huge machine, you run several medium ones in parallel with the same code: here, four with 96 vCPUs (about 9,600 req/s each).

Need more capacity? Add another server. There is no hard ceiling and each machine is only a slice of the total. This is horizontal scaling (scaling out), and it is how almost every large system grows.

What changed

  • 1 giant machine → 4 medium ones
  • Cost $7,200 → $14,400 a month
  • CPU ≈ 52% on each
  • Errors 100% → 0%

Step 7 of 11

Problem

Peak capacity, asleep

Traffic follows people's schedules. At 8 p.m. you need all four servers; at 4 a.m. there are a tenth of the users and each one runs at 5%.

You are paying for peak capacity 24 hours a day, even though most of the day it is barely used.

What changed

  • 4 a.m.: users 500,000 → 50,000
  • CPU ≈ 52% → 5%

Step 8 of 11

Solution

Let the fleet follow demand

Since all the servers are the same, an autoscaler can start and stop machines for you, for example to keep average CPU near 50%.

In the small hours it keeps two (the minimum, so it survives one going down) and in the evening it goes back to four. If four run half the day and two the other half, the average cost drops to $10,800 a month.

What changed

  • 2 servers stopped in the small hours
  • CPU 5% → 10% on the ones left
  • Average cost $14,400 → $10,800 a month

Step 9 of 11

Problem

Case: a lightning deal

At noon there are 150,000 users and two servers have room to spare (6,000 req/s, 31% CPU). A deal starts and within two minutes traffic grows 5×: 750,000 users, 30,000 req/s.

The autoscaler asks for 4 more servers, but in this example each takes about 3 minutes to boot. Meanwhile the two running ones serve 19,200 req/s: almost two thirds. The rest, 36%, fails. The fix is to scale ahead: schedule it for known events and keep headroom for surprises.

What changed

  • Users × 5 at noon
  • 4 servers booting (≈ 3 min)
  • Errors 0% → 36% while they boot

Step 10 of 11

Problem

So who sends each request to a server?

So far each group of users points at the address of one specific server. If Server B goes down, a quarter of the users get errors even though three healthy servers sit right next to it.

Clients do not know which servers are alive or saturated. Something has to sit in front, take every request and send it to a healthy server.

What changed

  • Server B down
  • Errors 0% → 25%

Step 11 of 11

Summary

Up versus out

Vertical: a bigger machine. No code changes and nothing new to run, but it has a ceiling, a restart at every jump and a single point of failure.

Horizontal: more identical machines. No practical ceiling, a failure only hits a slice and you can autoscale. In exchange, servers cannot keep state and you need a load balancer.

You will learn

  • Recognise when a server is saturated
  • Scale vertically and know where it ends
  • Scale horizontally with several identical machines
  • Autoscale with the daily rhythm and get ahead of peaks
  • Understand why scaling out needs a load balancer