Service discovery and gateways
- Lesson 29 of 40
- 13 min
- Intermediate
- 13 steps
In a microservices system machines change all the time. If one service needs another, how does it know which address to call today? And where do outside clients come in?
- Failed calls
- 0%
- Payments instances
- 3
- How Orders finds Payments
- mesh sidecars
- request
- response
- health check
- failed request
- event
Step 1 of 13
Concept
Orders calls Payments
With microservices, the Orders service charges customers through the Payments service. The most direct way: write the IPs of the two Payments machines into Orders' configuration and spread calls between them.
As long as those two machines stay the same, it works fine.
Step 2 of 13
Problem
Instances change on their own
In the cloud machines are replaceable: autoscaling adds and removes instances, every deploy renews them and failing ones get swapped out. Instance 10.0.1.14 was just replaced by another one at 10.0.2.31.
Orders' configuration never heard about it. Half of its calls go to an address where nobody answers anymore, and the new instance does not get a single one.
What changed
- 10.0.1.14 no longer exists
- 10.0.2.31 running, but Orders does not know it
Step 3 of 13
Solution
A service registry
Each instance signs itself into a service registry when it starts, then sends it a periodic heartbeat (the health checks in the drawing) to say it is still alive. Orders asks the registry where Payments is and gets the current list.
It is an address book that keeps itself up to date: nobody edits addresses by hand again.
What changed
- Service registry
- Instances register and send heartbeats
- Orders looks addresses up in the registry
Step 4 of 13
Concept
Instances join and leave on their own
Now churn does not hurt. A new instance starts at 10.0.4.7: it registers and gets traffic within seconds. The one at 10.0.2.31 hangs, stops sending heartbeats and, after three missed heartbeats (example threshold), the registry drops it from the list.
Nobody touched a single line of configuration.
What changed
- 10.0.4.7 registers
- 10.0.2.31 leaves the list
Step 5 of 13
Technique
Does the client pick, or a middleman?
Client-side discovery: Orders gets the list and picks the instance itself (in turns, least busy…). No extra hops, but every client, in every language, has to carry that logic.
Server-side discovery: Orders calls one fixed name and an internal load balancer queries the registry and forwards the call. Clients stay simple at the cost of one more hop.
What changed
- Picking the instance: in the client or in an internal load balancer
Step 6 of 13
Technique
Discovery through DNS
The registry can also answer over DNS: resolving payments.internal returns the IPs of the healthy instances. Every language and tool can already make a DNS query, no special libraries needed.
The price is caching: the operating system, the resolver and libraries keep the answer for its TTL, sometimes longer, and keep calling an instance that is gone. Use short TTLs and retry on connection errors.
What changed
- The registry answers DNS queries
- payments.internal → healthy IPs · TTL 10 s
Step 7 of 13
Problem
Traffic before it is ready
A new instance at 10.0.5.9 registers as soon as its process starts, before it opens database connections or warms its cache. DNS announces it right away and it gets its full share of traffic.
With three instances that is one call in three: while it finishes starting up (about 20 s in the example), that third times out. A live process is not the same as one that can serve.
What changed
- 10.0.5.9 starting up
- Gets traffic before it is ready
Step 8 of 13
Solution
Readiness, gradual start and draining
Split two questions: liveness («is the process working?») and readiness («can it serve now?»). The instance only joins the list once it passes its readiness check, and balancing raises its load little by little: slow start.
Shutting down is the reverse: first it deregisters, then it finishes the requests in flight (draining), and only then it stops.
What changed
- Readiness check before listing it
- Slow start: 10% to 100% over a minute (example values)
Step 9 of 13
Concept
What if the registry goes down?
Everything now depends on the registry, so it cannot be a single point of failure. First defence: each client keeps the last known list and keeps calling even when the registry does not answer; it just stops seeing changes for a while.
Second: the registry runs as a small cluster whose nodes decide by majority with a consensus algorithm such as Raft or Paxos. With 3 nodes, losing one changes nothing.
What changed
- The registry runs on 3 nodes
- One node down
- Clients carry on with their cached list
Step 10 of 13
Problem
Every team rewrites the same plumbing
Finding the instance is only the start. Every call between services also needs timeouts, retries, circuit breakers, encryption and metrics.
If each service is written in a different language, each team implements all of that its own way, with its own bugs and its own defaults.
What changed
- Retries, TLS and metrics inside every service
Step 11 of 13
Solution
A service mesh: the sidecar handles it
A service mesh puts a small proxy, the sidecar, next to every instance. The service only talks to its local sidecar, and the sidecars find instances, retry, apply timeouts, encrypt with mTLS and report metrics.
A control plane sends every sidecar the list of instances and the current policies. The plumbing is written once, in one place.
What changed
- A sidecar next to every service
- The registry becomes the control plane
- Plumbing leaves the services' code
Step 12 of 13
Technique
At the door: an API gateway
Discovery solves internal calls. For outside clients (apps and browsers) there is a single front door, the API gateway: it authenticates, applies rate limits, terminates TLS and sends each route to the right service.
Clients never learn the internal map, so you can split, merge or move services without breaking anything for them.
What changed
- API gateway in front of the services
- Clients only know the gateway
Step 13 of 13
Summary
Summary
Hand-written IPs go stale as instances come and go.
A registry with heartbeats, or served over DNS, keeps the list current.
An instance joins the list only when it is ready; before shutting it down, deregister it and drain it.
Discovery can live in the client, in an internal load balancer or in a mesh sidecar.
The API gateway is the single front door for outside clients.
← → step · R replay · Space pause
You will learn
- Why hand-written addresses break (covered)
- Service registries, heartbeats and DNS (covered)
- Client-side or server-side discovery (covered)
- Readiness, gradual start and draining (covered)
- A registry that survives losing a node (covered)
- Service meshes and API gateways (covered)