Replication

  • Lesson 12 of 40
  • 12 min
  • Beginner
  • 11 steps

A single database is both a bottleneck and a single point of failure. With several copies you spread the reads and survive losing a machine.

Leader CPU
30%
Machines reading
2
Replica lag
20 ms
Failed writes
0%
250,000 usersApp serverApp serverreplicaleaderreplicaanalytics replicabackups90 %reads + writes90 %10 %10 %copies every change30 %40 %40 %writes onlyname: Ana Ruizname: Ana Gómez · 2 s behindAna sees her old namename: Ana Ruizwrites + Ana's readsAna sees her new name40 %40 %downwrites fail30 %30 %promoted to leadercopying data30 %70 %finance · 20-min report100 %30 %40 %40 %status = 'cancelled' · 3M rowscopied the mistakecopied the mistakesnapshot 03:00 + logPoint-in-time recovery03:00restore the snapshot03:00–14:02replay the log14:02:59stop here: data intact14:03the broken migrationexample times30 %40 %40 %Ways to replicatesingle leaderone machine writessimplemulti-leaderone per regionconflictsleaderlessW + R > Nquorumone leader30 %40 %40 %
  • request
  • response
  • batch
  • replication

Step 1 of 11

Problem

One database for everything

Your app keeps everything on a single database server. Every read and every write goes through it, and it is already 90 % busy. Of those 90 points, 60 are reads (product pages, profiles, lists) and 30 are writes.

On top of that, if that machine goes down, the whole app stops working. The figures in this lesson are an example scenario.

Step 2 of 11

Solution

One leader and two replicas

Add two machines holding a copy of the data: the replicas (also called followers). Writes still go to a single machine, the leader.

Every change the leader applies is recorded in a replication log. The leader sends that log to the replicas and each one applies the same changes in the same order, so they end up with the same data.

What changed

  • Two replicas
  • The leader sends every change to the replicas
  • The app still asks the leader for everything

Step 3 of 11

Solution

Reads go to the replicas

Now the app sends writes to the leader and reads to the replicas. The 60 points of reads are split across two machines and the leader drops from 90 % to 30 %: what the writes cost.

Each replica sits at 40 %: 30 for reads and about 10 for applying the changes that arrive from the leader. Need more reads? Add another replica.

What changed

  • Reads → replicas
  • Writes → leader only
  • Leader CPU: 90 % → 30 %

Step 4 of 11

Problem

Replicas fall behind

Replication is usually asynchronous: the leader confirms the write as soon as it has stored it and sends it to the replicas right after.

Ana changes her name and reloads her profile. That read lands on a replica that has not received the change yet: she sees her previous name and thinks it was not saved.

What changed

  • Ana saves a new name
  • The replica still has the old one
  • Lag: 20 ms → 2 s

Step 5 of 11

Technique

Read what you just wrote

A practical fix: for a few seconds after someone writes, their reads go to the leader. Everyone else keeps reading from the replicas. That gives the read-your-own-writes guarantee.

The other route is synchronous replication: the leader waits for a replica to confirm the change before answering "saved". That replica is never behind, but every write takes longer and, if the replica does not answer, writes are left waiting.

What changed

  • Ana's reads → leader for 5 s
  • Leader CPU: 30 % → 31 %
  • Ana sees her new name

Step 6 of 11

Problem

The leader goes down

The leader's machine shuts down. Reads keep working, because the replicas have the data, but every write fails: nobody can sign up, buy or save changes.

The replicas drop to 30 %: they still serve reads, but no more changes arrive for them to apply.

What changed

  • Leader down
  • 100 % of writes fail

Step 7 of 11

Solution

Failover: a replica becomes the leader

Failover promotes the most up-to-date replica to leader, the app sends its writes there and the other replica starts copying from the new leader. A new machine replaces the dead one: it copies the data and, while it catches up, the only ready replica serves every read and climbs to 70 %.

Writes come back within seconds or a few minutes. Anything the old leader confirmed but never sent may be lost: another reason to keep one replica synchronous.

The hard part is being sure the leader is dead. If it was only slow or cut off from the network, you can end up with two leaders accepting writes: split brain. Avoiding it takes a majority agreeing; you will see how in the consensus lesson.

What changed

  • The dead leader leaves
  • A replica becomes the leader
  • New replica copying data
  • Remaining replica: 30 % → 70 %

Step 8 of 11

Problem

Case: the 20-minute report

Every month, finance runs a report that scans years of orders and takes 20 minutes. If it runs on a replica that also serves the app, that replica saturates, falls minutes behind and user reads turn slow.

The way out: an analytics replica the app never reads from. Heavy queries can run it at 100 % and take as long as they need without bothering anyone.

What changed

  • Analytics replica, outside the app's reads
  • The new replica now serves reads
  • Reports no longer slow the app

Step 9 of 11

Problem

Case: the migration without WHERE

A deploy runs a migration that is missing its WHERE: UPDATE orders SET status = 'cancelled'. The leader applies it to 3 million rows and replication copies the mistake to every replica in under a second.

Replicas protect you from a machine breaking, not from a wrong write. For that you need backups: a daily snapshot plus the archive of the change log. Together they can rebuild the database as it was at any moment: point-in-time recovery.

What changed

  • The migration overwrites 3M orders
  • The replicas copy the damage
  • Daily snapshot + log archive
  • You restore to 14:02:59, just before the mistake

Step 10 of 11

Trade-off

Three ways to replicate

Everything so far is single-leader replication: one machine writes and several read. It is the simplest, and the default in most relational databases.

With multiple leaders, each region has its own and writes stay close to the user, but if two regions change the same row at the same time there is a conflict to resolve.

With no leader, the client writes to W of the N copies and reads from R. If W + R > N, at least one of the copies read always has the latest write. You will see more in the CAP lesson.

Step 11 of 11

Summary

Summary

The leader takes the writes and sends them to the replicas through the replication log.

Replicas multiply reads and survive a machine going down.

Asynchronous replicas lag: right after writing, read from the leader.

If the leader dies, promote a replica and avoid ending up with two leaders.

Heavy queries go on their own replica.

A replica is not a backup: keep snapshots and the log, and rehearse restoring.

You will learn

  • Understand the leader, the replicas and the replication log
  • Spread reads across replicas
  • Handle lag and read what you just wrote
  • Promote a replica when the leader goes down
  • Keep analytics and backups apart, and pick a replication style