0% completed
How a Code Change Reaches Production
On This Page
From Commit to Merge
The Pipeline Stages
Build Once, Deploy the Same Image
Canary and Rollback
Infrastructure Changes Follow a Similar Path
The Path After Release
Using This in an Interview
Key Takeaways
Practice Questions
09:00 merge checkout-service commit 8f3a1c2 09:04 test passed 1,240 unit tests 09:06 image built, scanned checkout@sha256:4be19a 09:08 staging passed 12 smoke tests 09:18 canary passed errors 0.2% (current 0.2%), p95 410 ms (current 400 ms) 09:20 rollout complete 12 of 12 pods updated
These six lines record one small change to the grocery company's checkout service. A developer merged it at 09:00, and every customer was using it at 09:20. No person copied a file or logged in to a server.
Together, these lines show the path that a DevOps engineer builds and maintains.
From Commit to Merge
A commit is a saved set of code changes in version control, and a merge adds that work to the main branch. The main branch is the shared version of the code from which production releases are built.
Before the merge, another developer reviews the change, and automated checks run against it. Continuous integration (CI) is the practice of merging small changes into the main branch often and building and testing each one automatically. The 20-minute time in this lesson starts at the merge, after the review.
The Pipeline Stages
Each stage of the pipeline exists to catch a particular kind of problem as early and as cheaply as possible. Fast checks come first, because they stop many bad changes before slower stages begin.
| Stage | Minutes | Question it answers | Example of a problem it catches |
|---|---|---|---|
| Build and unit tests | 4 | Does the code compile and behave correctly in isolation? | A wrong tax calculation |
| Image build and scan | 2 | Can we package it, and does it contain known vulnerabilities? | An unsafe library version |
| Staging and smoke tests | 2 | Does it start and work with the services it needs? | A missing configuration value |
| Canary | 10 | Does it behave well with real traffic? | A slow database query |
| Full rollout | 2 | Can every running copy move safely to the new version? | A new copy that never becomes ready |
A unit test checks one small piece of code. Staging is a production-like environment that real customers do not use, and a smoke test is a short check that the most important functions work.
Build Once, Deploy the Same Image
The pipeline builds one container image and moves that exact image through every later stage. An image is the packaged, read-only set of files from which containers start. The log identifies it by a digest, here beginning sha256:4be19a, which is a value calculated from the image's contents. A version label, called a tag, can be moved to another image, but a digest changes whenever the contents change.
If the pipeline rebuilt the code for production, a library that the code uses might change between builds. The version tested in staging would then differ from the version that customers receive. Deploying by digest proves that both environments ran the same contents, and only configuration values differ between them.
Canary and Rollback
The canary tests the new version with a small share of real traffic before most customers receive it. Checkout runs as 12 pods, the smallest units that Kubernetes starts and manages, each holding one or more containers. During the canary, a traffic router sends 5 percent of checkout traffic to the new version for 10 minutes.
Checkout receives far more requests than orders, because shoppers view carts, choose delivery times, and retry payments. At the peak of about 2,000 requests a minute, 5 percent is 100 requests a minute, or about 1,000 during the canary. A service with little traffic would need a longer canary or a larger share to collect enough requests.
The pipeline compares the canary with the current version throughout those 10 minutes, using two rules:
- The error rate must not be meaningfully higher than the current version's rate.
- The p95 latency must not be more than 20 percent above the current version's p95.
In the log, the canary's p95 is 410 ms against 400 ms. The limit is 480 ms, so the change continues.
If either rule fails, the pipeline performs a rollback, which returns all traffic to the previous version. The old version is still running during the canary, so moving the traffic back takes only seconds. The whole recovery takes about 5 minutes from the start of the problem, because most of that time is needed to collect enough requests for a reliable comparison.
The final stage is a rolling update. Kubernetes replaces the pods a few at a time: it starts new pods, waits until they report that they are ready, and then removes old pods. Checkout therefore continues to serve customers throughout the change. Because no person approves any step after the merge, the practice is called continuous deployment.
Infrastructure Changes Follow a Similar Path
An infrastructure change also moves through review and automation, but one additional step shows its effect first. Suppose that an engineer proposes a larger database. terraform plan lists what Terraform would create, change, or destroy, reviewers read that list, and after approval terraform apply carries out the plan.
This preview matters because a small edit can have a large effect. If the plan says that the database will be replaced rather than resized, the engineer can stop before any data is at risk.
The Path After Release
Delivery does not end when the rollout finishes. Monitoring continues to compare the service with its SLO, and an alert reaches the on-call engineer if the service becomes unhealthy. After an incident, the team adds a test, an alert, or a pipeline rule so that the same failure is caught earlier next time.
The latency rule in this pipeline came from exactly this kind of review.
Using This in an Interview
"Walk me through what happens from a code push to production" tests how much of the path you understand. Follow one change, and explain why each stage exists.
"After review and merge, the pipeline runs unit tests, builds one image, scans it, and tests that same image in staging. Then 5 percent of real traffic goes to a canary for 10 minutes. If errors and p95 stay within limits, Kubernetes performs a rolling update; otherwise, traffic returns to the old version."
Expect two follow-up questions:
- "What if the release includes a database change?" Make the database change compatible with both versions, so that the old version still works during a rollback.
- "Why not run the canary for an hour?" A longer canary finds more problems but delays every deployment. The time should reflect the service's traffic and risk.
Key Takeaways
- Fast checks should run before slower and more realistic ones.
- The same image should move from testing to production.
- A canary limits harm by testing the change on a small share of real traffic.
- Incident reviews should improve the delivery path.
The next lesson, How DevOps and SRE Interview Loops Test the Job, connects this work to the rounds that companies use to assess it.
Practice Questions
1. Why does the pipeline run unit tests before the canary?
<details><summary>Show answer</summary>Because unit tests are fast and do not expose customers to the change. They stop simple mistakes within minutes. The canary is slower and uses real traffic, so it should receive only changes that have passed the earlier checks.
</details>2. The current p95 is 500 ms, and the canary's p95 is 590 ms. Does it pass the 20 percent rule?
<details><summary>Show answer</summary>Yes, because the limit is 600 ms. Twenty percent of 500 ms is 100 ms, so the canary may reach 600 ms. Its 590 ms is below that limit, although the error rate must also pass.
</details>3. A team rebuilds its image separately for staging and production. What risk does this create?
<details><summary>Show answer</summary>Production may run contents that staging never tested. A library or another build input can change between the two builds. Building once and deploying the same digest removes that difference.
</details>Reading Progress
0%
On This Page
From Commit to Merge
The Pipeline Stages
Build Once, Deploy the Same Image
Canary and Rollback
Infrastructure Changes Follow a Similar Path
The Path After Release
Using This in an Interview
Key Takeaways
Practice Questions