Observability

  • Lesson 27 of 40
  • 11 min
  • Intermediate
  • 9 steps

A system split across many services fails in ways nobody can see at a glance. Observability is what lets you ask it what is wrong and get an answer in minutes.

Time to notice
2 min
Time to the cause
5 min
p99 latency
430 ms
Signals
metrics + logs + traces
shoppersGatewayGatewayOrdersOrdersInventoryInventoryorders DBPaymentsPaymentsAnti-fraudexternalAnti-fraud????p99 10 msp99 40 msp99 2.5 sp99 2.4 sp99 2.6 sDashboard · p99 per servicegateway2.6 sorders2.5 spayments2.4 sinventory40 msorders DB10 msexample valuesSLO: burning 15× too fastp99 40 msp99 2.4 sCheckout SLOSLIcheckouts under 1 sSLO99.9% over 30 daysbudget0.1% may be slowlast hour1.5% slow = 15× → pageexample valuesreq=r7c22.2 s in a single callLogs · filter req=r7c214:02:12.204gateway: POST /checkout14:02:12.209orders: stock reserved14:02:12.251payments: calling anti-fraud14:02:14.449payments: anti-fraud ok (2198 ms)example times2.2 sTrace r7c2 · 2.6 sgateway2.6 sorders2.55 sinventory40 mspayments2.4 santi-fraud2.2 sexample valuestimeout 300 ms + cachep99 430 msTrace k4m9 · 430 msgateway430 ms› orders380 ms›› inventory40 ms›› payments230 ms››› anti-fraud30 ms (cached)example valuesconfig change 03:10errors 6%Failed checkouts · last hour02:550.1%03:050.1%03:10 config change0.1%03:126%03:17 reverted0.1%example valuesp99 430 msPost-mortem · checkout errorsimpact~6% of checkouts fail, 5 mincauseinventory timeout: 2 s → 20 mstimingpaged in 1 min · reverted 4 min lateractioncanary rollout for configactionvalidate timeouts before applyingp99 430 ms
  • request
  • response
  • slow request
  • failed request

Step 1 of 9

Problem

“Checkout takes forever.” Where?

Complaints arrive: paying for the cart takes several seconds. Every purchase goes through the gateway, the orders service, inventory, payments, a database and an external anti-fraud service.

In the diagram you can see a request stuck somewhere, but the team sees none of it. CPU looks normal on every server and nobody knows where to start looking.

Step 2 of 9

Technique

Metrics: numbers over time

Every service records a few numbers about the requests it serves, following the RED method: Rate (requests per second), Errors (share that fails) and Duration (latency percentiles). It ships them to a time-series database and a dashboard draws them.

With the dashboard in front of you the map changes: inventory and the database answer fast; the slowness lives on the payments path.

What changed

  • RED metrics on every service
  • Dashboard with each service’s p99

Step 3 of 9

Technique

Alert on what users notice

Waking someone because CPU went over 80% creates alerts at all hours that almost never mean anything. It is better to alert on symptoms users actually notice.

Pick an SLI (say, the share of checkouts that finish in under 1 s), set a target or SLO (99.9% over 30 days) and alert when you burn the remaining error budget too fast. Here, 1.5% slow checkouts in the last hour is 15 times the 0.1% allowed: page now.

What changed

  • SLO: 99.9% of checkouts under 1 s
  • Alert on error-budget burn

Step 4 of 9

Technique

Logs: what happened, line by line

Metrics tell you where; logs tell you what happened. Every service writes timestamped events, but at thousands of requests per second lines from different requests mix together and nothing makes sense.

The fix: the gateway assigns a request id, every service forwards it on its calls and writes it on every line. Filter by that id and you read the whole story of a single purchase.

What changed

  • Request id on every log line

Step 5 of 9

Solution

Traces: follow one request through the whole system

A distributed trace records every hop of a request as a span, with its start and duration, all tied together by the same trace id. Drawn as a waterfall, it shows exactly where the time goes.

In this purchase, 2.2 of its 2.6 seconds (about 85%) are spent waiting for the external anti-fraud service.

What changed

  • Trace id propagated across services
  • Cause: anti-fraud, ~85% of the time

Step 6 of 9

Solution

Fix it and prove it worked

With the cause in sight the fix is quick: call anti-fraud with a 300 ms timeout, cache recent scores and, if it does not answer in time, approve the payment and review it later.

The same dashboards confirm the result: p99 drops from 2.6 s to 430 ms and the SLO alert clears. Even when the cache misses and the full timeout is spent, the purchase stays under 1 s.

What changed

  • 300 ms timeout and cache on anti-fraud
  • Checkout p99: 2.6 s → 430 ms

Step 7 of 9

Problem

Case: errors jump at 3 a.m.

The alert fires: failed checkouts go from 0.1% to 6% at 03:12. The dashboard points at inventory, and the deploy log shows a configuration change to that service at 03:10: its timeout went from 2 s to 20 ms by mistake.

The right move is to mitigate first: revert the change and restore service. The underlying cause is investigated afterwards, calmly, not the other way round.

What changed

  • Config change to inventory at 03:10
  • Failed checkouts: 0.1% → 6%
  • Reverted at 03:17

Step 8 of 9

Concept

Learn from it: the post-mortem

Once the incident is over, the team writes a post-mortem: the timeline, the impact, the root cause and the actions that stop it from happening again.

It is blameless: the question is how the system let the mistake through, not who made it. Here, configuration changes now roll out in stages (canary), just like code.

What changed

  • Post-mortem written
  • Action: canary rollout for config too

Step 9 of 9

Summary

Recap

RED metrics (rate, errors, duration) show where it hurts.

Alert on SLOs, what users notice, not on CPU.

Logs with a request id tell the story of one request.

Traces follow a request across every service.

In an incident, mitigate first; then write a blameless post-mortem.

Instrument before you need it: in the middle of an outage it is too late to add logs.

You will learn

  • Why “it is slow” is so hard to investigate
  • Metrics and dashboards with the RED method
  • Alerts on SLOs, not on CPU
  • Logs with a request id and distributed traces
  • How to run an incident and write the post-mortem