Grokking DevOps and SRE Interviews
Vote

0% completed

​

What a DevOps Engineer Does

What the Role Is

What the Engineer Built

How the Nine Months Were Spent

How the Work Is Measured

Why More Deployments Reduced Risk

What the Role Requires

Using This in an Interview

Key Takeaways

Practice Questions

Title image: one release every two weeks becomes about 25 small deployments each working day.
Title image: one release every two weeks becomes about 25 small deployments each working day.

An online grocery company has 60 software engineers, 14 services, and about 30,000 orders a day. A service is a separately running program with one job, such as checkout or delivery tracking. Every second Tuesday night, three engineers combine about 120 finished changes into one release and spend 4 hours putting it live.

About 1 release in 4 breaks something that customers can see, and the average repair takes 2 hours because any of the 120 changes could be the cause.

The company then hires its first DevOps engineer. Nine months later, teams release their own changes about 25 times each working day. A finished change reaches every customer in about 20 minutes, and a bad change is removed automatically in about 5 minutes. What did this engineer build, and how does the company know that it worked?

What the Role Is

A DevOps engineer builds and maintains the systems that move a finished change into production, show how it behaves there, and let the team reverse it if it fails. Production is the environment that real customers use.

The word DevOps began as a way of working, not as a job title. Development teams wrote software and operations teams ran it, and because the first group was rewarded for change and the second for stability, releases often became a conflict. DevOps gives both groups shared responsibility for the running system and uses automation to make change safer.

The grocery company's new engineer therefore does not release software for the developers, but builds a path that lets each team release its own work.

What the Engineer Built

The path has five parts, which together replace the manual shared release. It begins when a change is merged, or added to the team's shared code.

The delivery system: a merged change moves through the pipeline to a canary and then to production, on infrastructure defined as code, with monitoring feeding results back.
The delivery system: a merged change moves through the pipeline to a canary and then to production, on infrastructure defined as code, with monitoring feeding results back.
  • Infrastructure as code. Servers, networks, and databases are described in files, and a tool such as Terraform creates and updates them, so an infrastructure change is reviewed like a code change.
  • A delivery pipeline. This automated sequence builds, tests, and deploys each change. To deploy means to install a new version in an environment and start it.
  • Containers and Kubernetes. A container packages a program with the libraries and files it needs to run, and Kubernetes starts containers across many machines and replaces those that fail.
  • Canary releases. A canary release sends a small share of traffic to the new version first. Here, the new version receives 5 percent of traffic for 10 minutes while the pipeline compares it with the current version.
  • Monitoring and alerts. Measurements from each service show whether it is healthy, and an alert reaches the on-call engineer, the person responsible for responding during a set period.

How the Nine Months Were Spent

The engineer did not replace everything at once. Each stage produced a working improvement before the next stage began.

MonthsWorkResult
1 and 2Wrote down and scripted every manual release stepThe 4-hour release became repeatable
3 and 4Added infrastructure as code and a pipeline for one low-risk serviceOne team could deploy on its own
5 and 6Moved the first services into containers on KubernetesServices gained one standard way to run and deploy
7 to 9Added canary releases and monitoring, and helped the teams move all 14 servicesTeams deployed independently

How the Work Is Measured

A DevOps engineer is measured by how quickly and safely changes reach customers, not by the number of tools installed. DORA (DevOps Research and Assessment), a long-running research program on software delivery, popularized the four measures below.

MeasureWhat it meansBeforeAfter
Deployment frequencyHow often changes are deployed to productionEvery two weeksAbout 25 times a working day
Lead time for changesTime from merged change to productionAbout 1 week on averageAbout 20 minutes
Change failure rateShare of deployments that cause a failure in production1 in 41 in 40
Recovery timeTime to restore service after a failure2 hoursAbout 5 minutes

The old lead time follows from the schedule. Changes were merged evenly across the 10 working days between releases, so the average change waited 5 working days. This company counts lead time from the merge, although DORA counts from the commit, the moment the code is saved in version control.

Why More Deployments Reduced Risk

More deployments do produce more failures in total. Teams now split their work into smaller changes, so the company makes about 250 deployments in two working weeks. At 1 failure in 40, that is about 6 failures in the period, whereas the old process had only a 1 in 4 chance of one failure.

The difference is the size of each failure. Expected harm is the average number of failures multiplied by the harm from each one. Suppose that an old failure affected every customer for 2 hours. Over two weeks, the expected harm is 0.25 x 120 minutes, or 30 minutes of full outage.

Suppose also that the canary catches every new failure, so each one reaches 5 percent of traffic for about 5 minutes. The expected harm is then 6.25 x 5 x 0.05, or about 1.6 minutes.

Expected customer harm over two weeks falls from 30 minutes of full outage to about 1.6 minutes, even though the number of failures rises.
Expected customer harm over two weeks falls from 30 minutes of full outage to about 1.6 minutes, even though the number of failures rises.

The second figure is a best case, because a failure that the canary misses reaches every customer. The next lesson examines one such failure.

What the Role Requires

The work draws on six skills, and the rest of this course develops each one:

  • Systems knowledge: how Linux and networks behave.
  • Automation: writing scripts and small programs that replace manual work.
  • Infrastructure: cloud services and infrastructure as code.
  • Delivery: pipelines and safe release methods.
  • Reliability: monitoring, alerts, and incident response.
  • Communication: explaining risks and helping development teams use the systems.

Two mistakes are common. Some DevOps teams accept tickets and perform every deployment by hand, which recreates the old separation. Others install many tools without changing any of the four measures.

Using This in an Interview

When you describe your work, start with the delivery problem, then give your decisions and the measured result.

"Our shared release ran every two weeks, took 4 hours, and failed about 1 time in 4. I scripted it first, moved one low-risk service to a pipeline, and then added canary releases. Teams now deploy about 25 times a working day, and a failed change is removed in about 5 minutes."

Expect two follow-up questions:

  • "Why start with one service?" It tested the pipeline where a mistake would cost little, and it gave other teams a working example.
  • "Did more deployments create more failures?" Yes, but each failure that the canary caught reached fewer customers for less time, so total harm fell.

Key Takeaways

  • A DevOps engineer builds the path that lets teams release safely on their own.
  • Deployment frequency, lead time, change failure rate, and recovery time show whether that path works.
  • Small, frequent changes can produce more failures but less total customer harm.
  • Automation should be added in stages, starting with one low-risk service.

The next lesson, DevOps, SRE, Platform, and Production Engineers, compares the role with three titles that often describe similar work.

Practice Questions

1. Why did the old release take so long to repair when it failed?

<details><summary>Show answer</summary>

Because about 120 changes were released together. Any of them could have caused the failure, so engineers first had to find the responsible change. A deployment with one small change has one likely cause and can be reversed directly.

</details>

2. A team installs Kubernetes, but it still deploys once every two weeks and fails 1 time in 4. Has its delivery improved?

<details><summary>Show answer</summary>

Not by the measures given, because its deployment frequency and change failure rate are unchanged. A new tool has value only when it improves results for the team and its customers. The team should find what still forces changes to wait or fail.

</details>

3. Every deployment now requires a ticket for the DevOps team. What problem has returned?

<details><summary>Show answer</summary>

The separation between the people who write software and the people who release it has returned. Changes wait in a queue, and developers lose responsibility for the result. The DevOps team should provide a safe path that developers use directly.

</details>

Reading Progress

0%


Vote for new content

On This Page

What the Role Is

What the Engineer Built

How the Nine Months Were Spent

How the Work Is Measured

Why More Deployments Reduced Risk

What the Role Requires

Using This in an Interview

Key Takeaways

Practice Questions