Grokking Staff Engineer Interview
Vote

0% completed

​

Senior and Staff Engineers

What Stays the Same

Scope

Time Horizon

Work Done Through Other People

Comparing Senior and Staff Work

Using This in an Interview

Key Takeaways

Practice Questions

A monthly job at an online marketplace charges each seller a listing fee. One night the job runs twice, and 1,900 sellers are charged twice. Two engineers investigate the same incident.

The Senior engineer finds the cause within a few hours. The scheduler that runs the job retried it after a 30-minute timeout, while the first run was still working. The engineer adds a check so that the job skips any seller already charged that month, refunds the duplicate charges, and writes the incident report. The fix ships in two days, and this job can no longer charge a seller twice.

The Staff engineer reads the same report and asks a different question: how many other jobs can fail this way? The answer takes a week to establish. The company runs three schedulers, built by three different teams, which together run 42,000 jobs a day for 26 owning teams. Two of the three retry a job after a timeout, and last quarter produced 6 incidents from jobs that ran twice or did not run at all. Each incident belonged to a different team, so no team had seen the pattern.

The Staff engineer writes a one-page proposal, persuades the owners of all three schedulers to adopt one retry rule, and starts a plan to move every job onto a single scheduler. The work takes three quarters, and other engineers write most of the code.

Both engineers did excellent work, but they answered different questions, and the difference between those questions is the difference between the two roles. This lesson describes it in three parts: the scope of the problems, how long the decisions last, and how much of the work is done through other people.

What Stays the Same

A Staff engineer is still an individual contributor, which means an engineer who does not manage people. Staff engineers do not hire, run performance reviews, or set anyone's priorities. They still read code, review designs, debug production systems, and often write the hardest part of a change themselves.

Technical depth is the basis of the role rather than something the engineer leaves behind. Other engineers accept a Staff engineer's proposal largely because they trust that engineer's technical judgment, and that trust comes from work they have seen.

The company in the opening appears throughout this course. It is an online marketplace with 400 engineers in 45 teams. The Staff engineer works in its platform group, reports to a director, and owns no team.

Scope

Scope is the part of the organization whose results depend on an engineer's work. A Senior engineer's scope is usually one team's systems, and the problems usually arrive already named: a feature to build, a bug to fix, a service to scale. A Staff engineer's scope is several teams or a whole organization, and the problems often have no name and no owner until someone finds them.

The two investigations in the opening show the difference in numbers:

Senior investigationStaff investigation
Jobs examined142,000 a day
Teams affected126
Schedulers involved13
Incidents explained16 last quarter
The same incident, examined at two scopes.
The same incident, examined at two scopes.

Notice that the Staff problem could not be seen from inside any single team. Each affected team saw one incident in a quarter and treated it as an isolated failure. Problems at this scope tend to share three properties:

  • They cross team boundaries. The cause sits in shared infrastructure, in a shared rule, or in the gap between two systems.
  • They have no single owner. Each scheduler team owned its own scheduler, but nobody owned the retry behavior across all three.
  • Their cost is spread thin. Six incidents divided among six teams never reach the top of any one team's priorities.

Finding such a problem is part of the job, and it is often the hardest part. A problem of this size is rarely assigned to anyone, because nobody above the teams can see it either.

Time Horizon

The time horizon of a decision is how long its effects last before the decision is changed or removed. Senior decisions often have a horizon of weeks or a few quarters. The check in the fee job matters to one job, and one engineer could rewrite it in an afternoon. Staff decisions often have a horizon of years, because many other decisions are built on top of them.

Here is a simple example. Suppose the new retry rule says that every job must be idempotent, meaning that a second run has no further effect. A job that cannot be made idempotent must declare this, and the scheduler then never retries it. Within a year, thousands of jobs will be written to that rule. If the rule later turns out to be wrong, each of the 26 teams must audit its jobs again. At 3 engineer-days per team, the reversal costs 78 engineer-days, against one afternoon for changing the check in one job.

The cost of reversing a decision grows with the number of people and systems that depend on it. This is why Staff engineers spend more time before deciding, write their decisions down with the reasons, and say which parts of a plan are deliberately temporary. The lesson Reversible and Irreversible Decisions shows how to match the effort spent on a decision to its cost of reversal.

The horizon also changes what "done" means. For the Senior engineer, the work was done when the fix was deployed and the duplicate charges were refunded. For the Staff engineer, the work is done when the whole class of problem stops recurring. That can only be measured over several quarters, as incidents from scheduled jobs fall from 6 a quarter to 1 or fewer.

Work Done Through Other People

A Staff engineer's results are measured mostly by what other engineers achieve because of that engineer's work. In the scheduler project, the Staff engineer personally wrote perhaps a tenth of the code. The platform team built the single scheduler, and the 26 owning teams moved their own jobs onto it. The Staff engineer's own work was different in kind:

  • Finding the pattern across six incident reports.
  • Writing the proposal and the retry rule.
  • Getting the three scheduler owners and the platform director to agree.
  • Moving the first team's jobs together with that team, so the other 25 teams had a working example to copy.
  • Reviewing the new scheduler's design, and helping teams whose migration had stalled.
How the Staff engineer's work becomes results across 26 teams.
How the Staff engineer's work becomes results across 26 teams.

Notice that the Staff engineer does the first steps and then appears only at a few points, while most of the effort happens in other teams. Over the three quarters, the Staff engineer spent about 40 engineer-days on the project, and the platform team and the owning teams spent about 300. None of those 300 days would have been spent on this problem without the first 40, because no single team could see the problem.

The value of that effort is also measurable. Suppose each incident costs about 4 engineer-days of response, investigation, and repair, before any refunds to customers. Six incidents a quarter cost about 24 engineer-days, and one incident a quarter costs about 4, so the project recovers about 20 engineer-days every quarter for as long as the rule stays in place.

Two failures are common at this level:

  • Doing all of the technical work personally. A Staff engineer who writes every migration becomes the slowest part of the project, and 26 teams wait for one person.
  • Doing none of the technical work. A Staff engineer who only writes documents gradually loses the technical judgment that made those documents worth reading.

The right balance between these differs from one Staff role to another, and the next lesson describes four common shapes of the role.

Comparing Senior and Staff Work

Senior engineerStaff engineer
Where problems come fromUsually assigned, or found within the teamOften found by the engineer, with no owner yet
ScopeOne team's systemsSeveral teams or an organization
How long decisions lastWeeks to a few quartersOften years
What "done" meansThe change is deployed and worksThe class of problem stops recurring
How results are measuredWhat the engineer deliversWhat other teams deliver because of the engineer's work
Typical written outputA design for a feature or serviceA proposal, a rule, a migration plan, or a strategy

A Staff engineer is therefore not simply a faster or more experienced Senior engineer. A Senior engineer who writes better code in less time is still working at Senior scope. The difference lies in which questions the engineer chooses to answer, and in how far the answers reach.

Using This in an Interview

Staff interviewers listen for scope, time horizon, and work through others in every answer, including system design answers. When you describe a project, state the scope in numbers, say how long the decision has lasted, and separate what you did yourself from what others did because of your work. For the scheduler project, a short answer covers all three:

"The bug was in one job, but two of our three schedulers retried after a timeout. That put 42,000 jobs a day across 26 teams at risk. I wrote the retry rule that every job still follows, and the teams moved their own jobs using an example I built with the first team."

Expect two follow-up questions:

  • "What did you do personally?" Name the proposal, the agreement, the first migration, and the reviews, and be precise about what the other teams built.
  • "How did you know the project was worth three quarters?" Give the incident count, the cost of each incident, and the reduction you expected.

Key Takeaways

  • A Staff engineer is still an individual contributor, and technical depth remains the basis of the role.
  • Staff scope is several teams or an organization, and its problems often have no name and no owner until the engineer finds them.
  • Staff decisions often last for years because other decisions are built on them, so they cost more to reverse and deserve more care.
  • A Staff engineer's results come mostly through other people and are measured by what other teams achieve.
  • The difference between the roles lies in the questions an engineer chooses to answer, not in how fast the engineer works.

The next lesson, The Four Staff Archetypes, describes the common shapes the Staff role takes and how each one divides the engineer's time.

Practice Questions

1. An engineer rewrites their team's service and cuts its response time from 400 ms to 90 ms. The work is difficult and takes a full quarter. Is this Staff-level work?

<details><summary>Show answer</summary>

Not on that evidence alone. Difficulty is not the same as scope. The work stays inside one team's systems, and its effects are measured in one service. It would become Staff-level work if, for example, 30 teams depended on that service and the engineer also set a response time budget that those teams adopted.

</details>

2. Why had no team noticed the scheduler problem before the Staff engineer did?

<details><summary>Show answer</summary>

Because its cost was spread across six teams. Each team saw one incident in a quarter and treated it as an isolated failure. The pattern only appeared when someone read all six incident reports together.

</details>

3. The investigation behind the retry rule took a week, while the fix in the fee job took two days. Why does the rule deserve more care?

<details><summary>Show answer</summary>

Because it costs far more to reverse. Thousands of jobs across 26 teams will be written to the rule. Reversing it later would cost about 78 engineer-days of audits, while the check in the fee job could be rewritten in an afternoon.

</details>

4. A Staff engineer decides to write every team's migration personally, to make sure each one is done correctly. What goes wrong?

<details><summary>Show answer</summary>

The engineer becomes the slowest part of the project. Twenty-six teams wait for one person, and the teams never learn the rule they must follow when they write new jobs. A better plan is to build one migration together with the first team and let the other teams copy it.

</details>

Reading Progress

0%


Vote for new content

On This Page

What Stays the Same

Scope

Time Horizon

Work Done Through Other People

Comparing Senior and Staff Work

Using This in an Interview

Key Takeaways

Practice Questions