Grokking the Anthropic Forward Deployed Engineer Interview
Vote

0% completed

​

Building an Evaluation Set From Customer Data

Suppose the accounting firm's evaluation set held 10 past closes. The agent would make about 97 proposals, and at a 5 percent error rate about 5 of them would be wrong. One more wrong proposal would move the rate by a full point.

For a rate p measured over n items, a rough margin of error is 2 x sqrt(p x (1 - p) / n). For 97 proposals that is 2 x sqrt(0.05 x 0.95 / 97), or about 4.4 points. The true rate could be anywhere from about 1 to about 9 percent, and the firm's limit was 8.

With 120 closes and 1,162 proposals, the same formula gives about 1.3 points: a true rate between about 3

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content