Timeouts and retries

  • Lesson 26 of 40
  • 11 min
  • Intermediate
  • 11 steps

When a dependency slows down or fails, the way you call it decides whether the problem stays there or takes your whole system with it. The numbers in the scene are example values.

Charged on time
99%
Payment call p99
180 ms
Load on payments
1×
Busy threads (of 200)
30
shoppersOrdersOrdersBreakerBreakerPaymentsPayments100%95%answering in 30 s200/200 threads waitingno timeout18%45%often slowfallback: payment pendingtimeout 800 ms20%50%timeout 800 ms · 1 retry100%100%4× normal load200/200 threads waiting3 retries, no pause23%60%recoveringbackoff + jitterOne call, with backoffattempt 1timeoutretry 1+100 msretry 2+200 msretry 3+400 ms ✓800 ms timeout · no jitter3%downfallback: payment pendingOPEN · fails in 1 ms15%50%CLOSEDcharging pending orders15%50%charged onceIdempotency-Key: a7c2CLOSEDIdempotency-Key: a7c2attempt 1charges 40 USD, stores the resultresponse lostorders retriesattempt 2known key: same resultin the enda single chargeexample amount14%45%deadline 2 s · 1.2 s leftbudget: 1.2 sdeadline 10:30:02.000CLOSED25%85%slow againpayments pool 40/40 · rest freepayments pool: 40 threads15%50%CLOSED
  • request
  • response
  • slow request
  • failed request
  • health check

Step 1 of 11

Problem

A slow service drags another one down

The payments service usually answers in 50 ms, but today it takes 30 seconds. The orders service calls it and keeps waiting, with no time limit at all.

Every waiting call holds a thread. Orders has 200 and in under a minute all of them are stuck: it can no longer serve even the pages that never touch payments. A problem in one dependency turns into a full outage.

Step 2 of 11

Solution

Add a timeout and a fallback

Every outgoing call gets a timeout: if payments has not answered in 800 ms, orders stops waiting and frees the thread. In this example 90 % of calls arrive in time and the other 10 % are cut off.

Those 10 % get a fallback: the order is accepted as “payment pending” and charged a little later. Orders stays up for everything else.

What changed

  • 800 ms timeout on payment calls
  • Fallback: “payment pending”
  • Busy threads: 200 → 36
  • Charged on time: 2 % → 90 %

Step 3 of 11

Technique

Retry the passing glitches

Many failures last an instant: a lost packet, an instance restarting, a short spike. Try again a moment later and it usually works.

With one retry after the timeout, 8 out of 10 cut-off calls succeed on the second go: charged on time rises from 90 % to 98 % (90 + 10 × 0.8). The cost: payments gets 10 % more calls (1.1×) and p99 grows to about 950 ms (800 of waiting + about 150 for the second attempt).

What changed

  • One retry on timeouts and 503s
  • Charged on time: 90 % → 98 %
  • Load on payments: 1× → 1.1×

Step 4 of 11

Problem

Retries turn into a storm

Now payments is truly overloaded. Every client retries three times with no pause, so each failing request turns into four: payments gets 4 times its normal load.

It can never recover: the retries themselves keep it down, and your clients behave like a denial-of-service attack. Every attempt uses up its 800 ms, so p99 reaches 3.2 s (4 × 800 ms) and the 200 threads fill up again.

What changed

  • Payments overloaded
  • 3 immediate retries per failure
  • Load on payments: 1.1× → 4×

Step 5 of 11

Solution

Wait longer each time, with jitter

Before retrying, wait, and double the wait each time: 100 ms, then 200, then 400. That is exponential backoff. Add a random part (jitter) so thousands of clients do not retry at the same instant.

Cap the attempts and also the share of traffic that may be retries: with a retry budget of 10 %, load never goes above 1.1×. Payments gets room to recover.

What changed

  • Waits of 100 → 200 → 400 ms (+ jitter)
  • Retry budget: 10 %
  • Load on payments: 4× → 1.1×

Step 6 of 11

Technique

Circuit breaker: stop calling for a while

Payments goes down completely. Even with a timeout, every call wastes 800 ms finding out. A circuit breaker counts failures: past a threshold it opens and rejects calls instantly, without trying them.

Every so often (here, 30 s) it lets one trial call through: that is the half-open state. If it succeeds, it closes again. Meanwhile, orders keep coming in as “payment pending”.

What changed

  • Payments down
  • Circuit breaker: OPEN
  • A failed call takes 1 ms, not 800 ms

Step 7 of 11

Solution

Half-open, then closed

Payments recovers. The next trial call succeeds, the breaker closes and traffic comes back gradually, so the service is not crushed the moment it returns.

The “payment pending” orders are charged from a queue. None was lost and the rest of the site never stopped working.

What changed

  • Payments recovered
  • Breaker: HALF-OPEN → CLOSED
  • Pending orders charged

Step 8 of 11

Technique

A retry must not charge twice

A tricky case: payments charges the card, but the response gets lost on the way back. Orders hits its timeout, retries… and the customer pays twice.

The fix is an idempotency key: a unique id per order that travels with every attempt, for example in an Idempotency-Key header. Payments stores the result for each key and, when a duplicate arrives, returns that result instead of charging again.

What changed

  • Idempotency-Key on payment calls
  • A repeated attempt returns the stored result

Step 9 of 11

Technique

Pass the deadline down

The shopper's browser gives up after 2 seconds. If orders has already spent 1.9 s, there is no point in payments starting 800 ms of work: nobody will see the answer.

So every request carries a deadline: “answer before 10:30:02.000”. Each service checks how much time is left and passes that budget on; with nothing left, it stops right away. In the scene, orders spent 0.8 s and payments has 1.2 s left.

What changed

  • Deadline on every request
  • Nothing starts if it cannot finish in time

Step 10 of 11

Technique

Bulkheads: one pool per dependency

Remember step 1, when slow payments grabbed all 200 threads? Orders can also protect itself by giving each dependency its own pool: for example, 40 threads for payments, 40 for inventory and 120 for everything else.

Now, when payments turns slow, only its 40 threads fill up. About 50 are busy in total (40 for payments and about 10 for the rest), and browsing, search and the cart keep working. It is the idea behind a ship's watertight compartments: the bulkhead pattern.

What changed

  • One thread pool per dependency
  • Slow payments fills only its 40 threads
  • The rest of the site is unaffected

Step 11 of 11

Summary

Summary

Every remote call needs a timeout and a fallback.

Retry only passing errors, with exponential backoff, jitter and a cap.

A circuit breaker stops hammering a service that is down.

Anything that gets retried must be idempotent.

Propagate deadlines and isolate dependencies with bulkheads.

You will learn

  • How one slow service spreads failure
  • Timeouts and fallbacks
  • Retries, retry storms and backoff with jitter
  • Circuit breakers and idempotency keys
  • Deadlines and bulkheads