Service discovery and gateways

  • Lesson 29 of 40
  • 13 min
  • Intermediate
  • 13 steps

In a microservices system machines change all the time. If one service needs another, how does it know which address to call today? And where do outside clients come in?

Failed calls
0%
Payments instances
3
How Orders finds Payments
mesh sidecars
OrdersOrdersOrders +sidecarOrders + sidecarPayments10.0.1.12Payments10.0.1.12Payments +sidecarPayments + sidecarPayments10.0.1.14Payments10.0.1.14Payments10.0.1.14Payments 10.0.1.14Payments10.0.2.31Payments10.0.2.31Payments10.0.2.31Payments 10.0.2.31Payments10.0.4.7Payments10.0.4.7Payments +sidecarPayments + sidecarPayments10.0.5.9Payments10.0.5.9Payments +sidecarPayments + sidecarRegistryRegistryRegistryRegistryDNS · TTL10 sDNS · TTL10 sDNS · TTL10 sDNS · TTL 10 sControlplaneControlplaneControlplaneControl planeregistry 2registry 3APIgatewayAPIgatewayAPIgatewayAPI gatewayapps and browsersconfig: 10.0.1.12, 10.0.1.14replacedidle · nobody knows itconfig: 10.0.1.12, 10.0.1.14cached list · 10 spayments → .1.12, .2.31where is payments?cached list · 10 s3 missed heartbeatsnew · registeredpayments → .1.12, .4.7where is payments?where is payments?Who picks the instance?client-sideOrders balances with the listserver-sidea load balancer picks for itresolves payments.internalpayments.internal?just started · coldtimes outready · ramping 10% to 100%cached list · still servingleader · 2 of 3 updownwhere is payments?retries · TLS · metrics · v1retries · TLS · metrics · v2sends instances and rulesauth · limits · routing
  • request
  • response
  • health check
  • failed request
  • event

Step 1 of 13

Concept

Orders calls Payments

With microservices, the Orders service charges customers through the Payments service. The most direct way: write the IPs of the two Payments machines into Orders' configuration and spread calls between them.

As long as those two machines stay the same, it works fine.

Step 2 of 13

Problem

Instances change on their own

In the cloud machines are replaceable: autoscaling adds and removes instances, every deploy renews them and failing ones get swapped out. Instance 10.0.1.14 was just replaced by another one at 10.0.2.31.

Orders' configuration never heard about it. Half of its calls go to an address where nobody answers anymore, and the new instance does not get a single one.

What changed

  • 10.0.1.14 no longer exists
  • 10.0.2.31 running, but Orders does not know it

Step 3 of 13

Solution

A service registry

Each instance signs itself into a service registry when it starts, then sends it a periodic heartbeat (the health checks in the drawing) to say it is still alive. Orders asks the registry where Payments is and gets the current list.

It is an address book that keeps itself up to date: nobody edits addresses by hand again.

What changed

  • Service registry
  • Instances register and send heartbeats
  • Orders looks addresses up in the registry

Step 4 of 13

Concept

Instances join and leave on their own

Now churn does not hurt. A new instance starts at 10.0.4.7: it registers and gets traffic within seconds. The one at 10.0.2.31 hangs, stops sending heartbeats and, after three missed heartbeats (example threshold), the registry drops it from the list.

Nobody touched a single line of configuration.

What changed

  • 10.0.4.7 registers
  • 10.0.2.31 leaves the list

Step 5 of 13

Technique

Does the client pick, or a middleman?

Client-side discovery: Orders gets the list and picks the instance itself (in turns, least busy…). No extra hops, but every client, in every language, has to carry that logic.

Server-side discovery: Orders calls one fixed name and an internal load balancer queries the registry and forwards the call. Clients stay simple at the cost of one more hop.

What changed

  • Picking the instance: in the client or in an internal load balancer

Step 6 of 13

Technique

Discovery through DNS

The registry can also answer over DNS: resolving payments.internal returns the IPs of the healthy instances. Every language and tool can already make a DNS query, no special libraries needed.

The price is caching: the operating system, the resolver and libraries keep the answer for its TTL, sometimes longer, and keep calling an instance that is gone. Use short TTLs and retry on connection errors.

What changed

  • The registry answers DNS queries
  • payments.internal → healthy IPs · TTL 10 s

Step 7 of 13

Problem

Traffic before it is ready

A new instance at 10.0.5.9 registers as soon as its process starts, before it opens database connections or warms its cache. DNS announces it right away and it gets its full share of traffic.

With three instances that is one call in three: while it finishes starting up (about 20 s in the example), that third times out. A live process is not the same as one that can serve.

What changed

  • 10.0.5.9 starting up
  • Gets traffic before it is ready

Step 8 of 13

Solution

Readiness, gradual start and draining

Split two questions: liveness («is the process working?») and readiness («can it serve now?»). The instance only joins the list once it passes its readiness check, and balancing raises its load little by little: slow start.

Shutting down is the reverse: first it deregisters, then it finishes the requests in flight (draining), and only then it stops.

What changed

  • Readiness check before listing it
  • Slow start: 10% to 100% over a minute (example values)

Step 9 of 13

Concept

What if the registry goes down?

Everything now depends on the registry, so it cannot be a single point of failure. First defence: each client keeps the last known list and keeps calling even when the registry does not answer; it just stops seeing changes for a while.

Second: the registry runs as a small cluster whose nodes decide by majority with a consensus algorithm such as Raft or Paxos. With 3 nodes, losing one changes nothing.

What changed

  • The registry runs on 3 nodes
  • One node down
  • Clients carry on with their cached list

Step 10 of 13

Problem

Every team rewrites the same plumbing

Finding the instance is only the start. Every call between services also needs timeouts, retries, circuit breakers, encryption and metrics.

If each service is written in a different language, each team implements all of that its own way, with its own bugs and its own defaults.

What changed

  • Retries, TLS and metrics inside every service

Step 11 of 13

Solution

A service mesh: the sidecar handles it

A service mesh puts a small proxy, the sidecar, next to every instance. The service only talks to its local sidecar, and the sidecars find instances, retry, apply timeouts, encrypt with mTLS and report metrics.

A control plane sends every sidecar the list of instances and the current policies. The plumbing is written once, in one place.

What changed

  • A sidecar next to every service
  • The registry becomes the control plane
  • Plumbing leaves the services' code

Step 12 of 13

Technique

At the door: an API gateway

Discovery solves internal calls. For outside clients (apps and browsers) there is a single front door, the API gateway: it authenticates, applies rate limits, terminates TLS and sends each route to the right service.

Clients never learn the internal map, so you can split, merge or move services without breaking anything for them.

What changed

  • API gateway in front of the services
  • Clients only know the gateway

Step 13 of 13

Summary

Summary

Hand-written IPs go stale as instances come and go.

A registry with heartbeats, or served over DNS, keeps the list current.

An instance joins the list only when it is ready; before shutting it down, deregister it and drain it.

Discovery can live in the client, in an internal load balancer or in a mesh sidecar.

The API gateway is the single front door for outside clients.

You will learn

  • Why hand-written addresses break
  • Service registries, heartbeats and DNS
  • Client-side or server-side discovery
  • Readiness, gradual start and draining
  • A registry that survives losing a node
  • Service meshes and API gateways