From a Vague Ask to a Problem Statement

"Scheduled jobs keep failing. Fix it."

The VP of Engineering at the marketplace sent that message to the director of the platform group after a quarterly review, and the director passed it to the platform Staff engineer. It contains four words of complaint and two words of instruction. It names no job, no team, no number, and no date.

A request of this kind is normal at Staff level. The people who see a problem first are often far from its cause, and they describe what they experienced rather than what is wrong. Before anyone designs anything, someone has to turn the complaint into a statement that several teams can read, check, and accept. This lesson answers how that statement is built, and how to tell a finished statement from a first draft.

What a Problem Statement Is

A problem statement is a short written description of a problem that says who is affected, how often, what it costs today, what success means in numbers, which constraints apply, and who must agree. It does not describe the solution. Its purpose is to let people who disagree about the solution first agree on the problem.

The difference from an ordinary requirement is the audience. A requirement is usually written for one team that will build something. A Staff-level problem statement is written for several groups at once: the person who asked, the teams who own the affected systems, and the teams who will pay for the change. Each group reads it to answer one question: "Is this the problem I think we have?"

Symptoms and Causes

A symptom is what someone observes; a cause is the mechanism that produces it. The VP observed a symptom: scheduled jobs failing. The word "failing" covers several different events, and the first task is to list them.

The Staff engineer read the six incident reports from the last quarter and sorted them by what actually happened:

The table shows something the VP's sentence hides. The six incidents come from two opposite failures. Schedulers A and B retry too eagerly, so a slow job runs twice. Scheduler C never retries, so a job whose run fails is not run again, and in one case nobody noticed for three days.

The opposite failures matter for the design. The obvious fix for double runs is to turn retries off, but that would turn A and B into copies of C and produce missed runs instead. The obvious fix for missed runs is to turn retries on, which would produce double runs on C. No single scheduler setting removes both failures. The cause is a missing piece of information about the jobs: the scheduler cannot know whether a given job is safe to run twice, because nobody has told it.

Notice that the VP's single word "failing" divides into two failures with opposite fixes. Both lead back to the same missing piece of information: whether each job is safe to run twice.

Questions to Ask the Requester

The requester knows what they saw and why it mattered; the Staff engineer needs both before writing anything. In a conversation of about 20 minutes, the VP answered most of what the incident reports could not. The Staff engineer asked five questions:

• "Which failures did you hear about?" The VP had heard about two: the seller fee job that charged 1,900 sellers twice, and a nightly report that was missing for three days.
• "What happened as a result?" Refunds, support tickets from sellers, and a finance team that no longer trusted the nightly numbers.
• "What would tell you this is fixed?" The VP's answer was "a quarter with no seller-facing incident from a scheduled job," which is stricter than an internal count and worth recording.
• "Is there a date this matters by?" Finance planned to add a second monthly billing job in two quarters, and wanted it to run on something reliable.
• "Who else should I talk to?" The VP named the finance lead and the payments team's engineering manager.

The questions ask about effects, not solutions. If the Staff engineer had asked "Should we build one scheduler?", the VP might have said yes. The work would then have started from a solution that nobody had tested against the problem.

Advanced System Design Fundamentals teaches a related skill for a stated design question in its lesson The First Ten Minutes. The work here starts one step earlier, before anyone has stated a design question at all.

The First Draft

The Staff engineer's first draft, written after reading the incident reports and before talking to anyone, was one sentence:

Replace the three job schedulers with one reliable scheduler.

It is a reasonable instinct and a weak problem statement, for four reasons:

• It names a solution. One scheduler may well be the answer, but the statement removes the chance to compare it with cheaper options, such as fixing retries in each scheduler separately.
• "Reliable" is not measurable. Nobody can check it, so nobody can say when the work is finished.
• It names nobody who is affected. The 26 owning teams and the sellers do not appear.
• It hides the cost to others. Replacing schedulers A and B means the payments team and the data team give up systems they own, and the statement does not say so.

The Agreed Version

A problem statement is finished when every group that must agree has read it and accepted it, not when its author is satisfied. The agreed version took two weeks and three rounds of comments. It is longer than the draft, and every added line answers one of the four weaknesses above.

Problem. Scheduled jobs at the marketplace run on three schedulers owned by three teams: A (payments, 18,000 jobs a day), B (data, 16,000), and C (platform, 8,000). Together they serve 26 owning teams. Last quarter, 4 jobs ran twice because A and B retried them after a timeout, and 2 jobs did not run because C does not retry. One double run charged 1,900 sellers twice.

Cause. No scheduler knows whether a job is safe to run twice, and no team owns the retry behavior across all three.

Cost today. About 24 engineer-days a quarter in incident response, plus refunds and seller support.

Success. At most 1 incident a quarter from jobs that run twice or not at all, and none that affects sellers, for two quarters in a row. Every missed run is detected within 15 minutes. Every job declares whether it is safe to retry.

Constraints. The monthly seller fee job must not miss a run. Each owning team spends about 5 engineer-days or less on average. Finance's new billing job must have a reliable home in two quarters.

Must agree. The VP of Engineering, the payments and data team managers, the director of the platform group, and the finance lead.

The success line uses three measures because each one checks something different. The incident count checks the result. The 15-minute detection target catches missed runs that would otherwise wait days to be found. The declaration requirement checks that the cause has been removed rather than hidden.