Grokking the Forward Deployed Engineer Interview
Vote

0% completed

​

Defining Success Before Building

Why Agree First

Three Kinds of Measures

Measuring the Baseline

Setting the Targets

The Tradeoff Between Two Numbers

Checking Success in the Pilot

Using This in an Interview

Key Takeaways

Practice Questions

Title image: three agreed targets for the routing system, each with the result it reached.
Title image: three agreed targets for the routing system, each with the result it reached.

Suppose the routing system sends 70 percent of claims to a queue automatically and is right on 97 percent of those. That is 840 automatic claims a day, and 3 percent of 840 is about 25 claims in the wrong queue with no person checking.

Is that a success? It depends on what the claims manager expected, what people achieved before, and which queues those 25 claims went to.

Why Agree First

Success must be defined in numbers, and agreed with the sponsor, before the system is built, while nobody has a result to defend. After the system exists, every result can start a new argument: a sponsor who hoped for 90 percent automation is disappointed by 70. Agreeing first also tells the adjusters, who saw the demo misroute 1 claim in 4, which error rate would stop the project.

Three Kinds of Measures

The FDE grouped the insurer's measures into three kinds, and a project needs all three.

  • Outcome measures describe what changes for the business. For the insurer: adjuster-hours spent on routing each day, and the time a claim waits to reach its queue.
  • System measures describe how well the system does its own job. For the insurer: the share of claims routed automatically, called the automation rate, and the share of automatic decisions that match the correct queue, called the agreement rate.
  • Guardrails are limits that must always be met, even if every other number looks good. For the insurer: possible fraud is never routed automatically, and every automated decision is written to the decision log.
The three kinds of measures for the routing system, with the agreed target and the measured result for each.
The three kinds of measures for the routing system, with the agreed target and the measured result for each.

Routing every claim automatically saves the most hours but routes possible fraud without an adjuster. That fails a guardrail, and no number of saved hours can justify it.

Measuring the Baseline

A target means little without a starting point, called the baseline. The FDE measured three baselines in the first week.

The adjusters timed themselves on 300 claims at about 5 minutes each, which is 100 adjuster-hours a day. Claims system timestamps showed an average wait of 2 days to reach the final queue.

The queue history showed about 8 percent of past claims rerouted, so the adjusters' own first choice was wrong about 8 percent of the time. That gave the agreement rate a fair comparison.

Setting the Targets

The FDE proposed targets in the first week, and the claims manager agreed to them in writing.

MeasureKindBaselineTargetMeasured how
Routing workOutcomeAbout 100 adjuster-hours a dayUnder 40Adjuster time records, after production
Time to the right queueOutcome2 daysUnder 1 hour for automatic claimsClaims system timestamps
Automation rateSystem0 percentAt least 60 percentTest set of 2,000 past claims
Agreement rateSystem92 percent (first choice, by people)At least 95 percentSame test set, against the final queue
Possible fraud never automaticGuardrailNot applicableZero automatic fraud decisionsDecision log
Every decision loggedGuardrailNo log existed100 percentDaily count of log entries against claims routed

The automation target of 60 percent was useful to the business but set below what the FDE expected, because the result was uncertain before the test set was run. A missed target damages trust more than a modest target that is passed. The agreement target of 95 percent sits above the 92 percent of people's first choices, so automatic decisions are at least as accurate.

The Tradeoff Between Two Numbers

The automation rate and the agreement rate depend on each other. The model gives each claim a confidence score, and a claim is routed automatically only when its score is above the queue's threshold. A lower threshold routes more claims automatically, but more of them are wrong. A higher threshold routes fewer, but more of them are right.

The FDE tried three settings on the 2,000 past claims:

SettingAutomation rateAgreement rateWrong automatic claims per day (of 1,200)
Low thresholds80 percent93 percent1,200 x 0.80 x 0.07 = about 67
Chosen thresholds70 percent97 percent1,200 x 0.70 x 0.03 = about 25
High thresholds55 percent99 percent1,200 x 0.55 x 0.01 = about 7
Three threshold settings tested on 2,000 past claims, showing that a higher automation rate brings more wrong automatic claims per day.
Three threshold settings tested on 2,000 past claims, showing that a higher automation rate brings more wrong automatic claims per day.

The low setting failed the agreement target, and the high setting failed the automation target. The chosen setting passed both. Because the targets existed before the test, the choice took one meeting rather than a debate. A target turns a tradeoff into a decision the customer can make.

Checking Success in the Pilot

The pilot must confirm the test set's numbers on new claims, because new claims can differ from past ones. During the pilot on glass and minor damage, adjusters checked 1 in 10 automatically routed claims each day.

Of about 600 pilot claims a day, about 420 were routed automatically, so adjusters checked about 42 a day. Ten working days gave about 420 checked claims, enough to see whether agreement stayed near 97 percent. It did, and all six queues were enabled in the sixth week.

Using This in an Interview

When asked how you would know your system works, give measures of all three kinds, each with a baseline and an agreed target.

"Before building, I would agree on outcomes, adjuster-hours and waiting time, and on the automation and agreement rates. The guardrails are that fraud is never automatic and every decision is logged. People's first choice is wrong about 8 percent of the time, so I would target 95 percent agreement."

Expect two follow-up questions:

  • "The customer wants 90 percent automation. What do you say?" Show that 80 percent automation meant about 67 wrong automatic claims a day, and offer to revisit the threshold after the pilot.
  • "Why not report one accuracy number for all claims?" A single accuracy number hides which claims are routed automatically and which queues the errors go to.

Key Takeaways

  • Agree on success in numbers with the sponsor before building, while no one has a result to defend.
  • Use outcome measures, system measures, and guardrails, each with a measured baseline.
  • A lower confidence threshold raises the automation rate and lowers the agreement rate.
  • Confirm the test set's numbers on new data in the pilot, with a sample checked by people.

The next lesson, The Smallest Version That Delivers Value, uses these measures to choose what to build and release first.

Practice Questions

1. Why was the automation target set at 60 percent when the FDE expected a higher result?

<details><summary>Show answer</summary>

Because a missed target damages trust more than a modest target that is passed. The result was uncertain before the test set was run, and a result of 70 percent passed it, which gave the claims manager confidence.

</details>

2. The system routes 80 percent of claims automatically with 93 percent agreement. How many wrong automatic claims is that per day, and does it meet the targets?

<details><summary>Show answer</summary>

About 67 a day, and it fails the agreement target. 1,200 x 0.80 = 960 automatic claims, and 7 percent of 960 is about 67. An agreement rate of 93 percent is below the 95 percent target.

</details>

3. Why is "possible fraud is never routed automatically" a guardrail and not a system measure?

<details><summary>Show answer</summary>

Because it is a limit that must be met no matter how good the other numbers are. A system measure can be traded against another measure, but a guardrail cannot.

</details>

Reading Progress

0%


Vote for new content

On This Page

Why Agree First

Three Kinds of Measures

Measuring the Baseline

Setting the Targets

The Tradeoff Between Two Numbers

Checking Success in the Pilot

Using This in an Interview

Key Takeaways

Practice Questions