CI/CD and deployments

  • Lesson 36 of 40
  • 13 min
  • Intermediate
  • 11 steps

Getting every change to production fast and calmly, and being able to undo it just as fast when something goes wrong.

Deploys
15/day
Commit → production
3 h
Deploys that fail
3%
Time to restore
1 min
developerProductionProductionusersGitmainGitBuildBuildTestsTestsartifactsStagingStagingProductionProductionv41v41v41v42v42v41v41v41v42v42v41v42v41v42v42v42v42v42v42v42LoadbalancerLoad balancerusersusersFlagsFlagsbuilt on my laptopcopied over SSH40%which of the 30 changes?DOWNsite downsmall changes, dailyclean machinetests greentests greentienda:a1c94e7smoke tests passedautomatic deploypromote · 1 clickgreen · v42 liveblue · v41 standbyall traffic → greencanary · errors 0.1%5% → v42no column phonephone → phone_numberExpand, then contract1 add phone_number, write to both2 copy the value into old rows3 read from phone_number4 drop phone once nothing uses itnever rename in a single stepone_click_pay: team + 1%Before → afterdeploys1/week → 15/daycommit → production2 weeks → 3 hdeploys that fail25% → 3%time to restore3 h → 1 minexample values
  • request
  • response
  • event
  • failed request

Step 1 of 11

Concept

Deploying by hand

Every Friday afternoon, someone on the team builds the app on their laptop, uploads the result to the production server over SSH and restarts it. Everyone else stays online in case something blows up.

That single deploy carries the whole week's changes together. When something fails, nobody can say for sure which of them is to blame.

Step 2 of 11

Problem

“Works on my machine”

This time the laptop had a newer version of a library than the server, and the app won't start. The site stays down while someone goes through about thirty changes looking for the one to undo.

Nobody wrote down which exact version was running before, so going back is a gamble too. The service returns three hours later.

What changed

  • The new version won't start on the server
  • Site down while people hunt for the guilty change

Step 3 of 11

Solution

Continuous integration

Now every git push to the shared repository triggers a pipeline on a clean machine: it builds the code and runs the tests. Nothing depends on anyone's laptop any more.

The team merges small changes into main several times a day, and a red pipeline gets fixed before any other task. That way main always builds and passes its tests: that is continuous integration (CI).

What changed

  • Shared repository and small changes on main
  • Every push is built and tested on a clean machine

Step 4 of 11

Technique

Build once, promote the same artifact

The last stage packages the result into an immutable artifact, almost always a container image, with a tag that points to the commit it came from (tienda:a1c94e7), and stores it in a registry.

That same image is the one that gets tested, the one that reaches staging and the one that ends up in production; only the configuration (addresses, secrets) changes between environments. And going back is as simple as deploying the previous image.

What changed

  • Packaging stage: one image per commit
  • Artifact registry

Step 5 of 11

Solution

Continuous delivery

Every image that passes the pipeline is deployed to staging on its own, a small replica of production, where smoke tests walk through the main paths. From there it can go to production whenever you want.

With continuous delivery, production is one click away. With continuous deployment there isn't even a click: every green change goes out by itself.

What changed

  • Automatic deploy to staging with smoke tests
  • One-click (or automatic) step to production

Step 6 of 11

Technique

Blue-green: switch groups instead of replacing

How does the new version reach users without cutting the service? In a blue-green deployment there are two identical production groups. Blue serves everyone on v41 while v42 starts up and warms up on green.

Then the load balancer sends all traffic to green in one go. If something looks off, going back to blue takes seconds: it is still running and untouched.

What changed

  • Two groups: blue (v41) and green (v42)
  • The load balancer moves all traffic to green

Step 7 of 11

Technique

Canary: a few users first

In a canary release, the load balancer sends a small share of real traffic, here 5%, to the new version, and the pipeline compares its error rate and latency with those of the previous version.

If v42 does worse, it is pulled back automatically and only a few users ever saw it. If it holds up, the share grows in stages, for example 25%, 50% and 100%.

What changed

  • 5% of traffic goes to v42
  • Automatic comparison with v41
  • Grows to 100% or is rolled back

Step 8 of 11

Problem

Case: the migration that breaks the previous version

v42 renames the phone column to phone_number. The migration runs and v42 works, but halfway through the deploy three of the four servers are still on v41, which asks for phone and gets an error.

The way out is to expand, then contract: add phone_number and write to both; copy the value into the old rows; switch reads to phone_number; and drop phone only when no running version uses it. Every step works for both versions.

What changed

  • Column renamed in a single step
  • Servers on v41 fail halfway through the deploy
  • Expand, migrate and contract, over several deploys

Step 9 of 11

Technique

Feature flags: deploying is not releasing

The new one-click payment has been in production for weeks, but switched off. A feature flag decides on every request who sees it: first the team, then 1% of users and finally everyone.

Switching it off is changing a setting, not deploying, so a problem is cut off in seconds. Deploying (moving code to the servers) and releasing (showing it to users) become separate decisions, and code can ship several times a day even when the feature isn't ready.

What changed

  • Flag service
  • One-click payment: on for the team and 1% of users

Step 10 of 11

Concept

Measure how you deliver

Four numbers, known as the DORA metrics, sum up how a team delivers software: how often it deploys, how long a change takes from commit to production, what share of deploys causes a failure and how long it takes to restore service when one does.

Studies across thousands of teams found what this lesson shows: those who deploy more often also break production less, because every change is small and every rollback is quick.

Step 11 of 11

Summary

Recap

CI: small changes merged often; every push is built and tested on a clean machine.

Build once: one immutable artifact per commit, promoted unchanged through every environment.

CD: every green change can reach production at any time, with a click or by itself.

Release with care: blue-green to go back fast, canaries to limit the damage and flags to separate deploying from releasing.

Schema changes must serve old and new code at the same time: expand, then contract.

You will learn

  • Build and test on every push
  • Build once and promote a single artifact
  • Release with blue-green and with canaries
  • Change the schema without breaking the previous version
  • Separate deploying from releasing and measure delivery