Grokking the Google Forward Deployed Engineer Interview
Vote

0% completed

​

Scaling Agents: Concurrency, Quotas, and Cost

The freight carrier's exceptions arrive Monday to Friday, about 900 a day. Monday is the peak at about 1,400, and its busiest hour brings about 220.

An average exception uses about 6 model requests, each with about 12,000 input tokens. For sizing, this lesson uses 8 requests per exception, which leaves a margin for longer runs. In the busiest hour, that is 220 × 8 = 1,760 requests, or about 29 a minute. It is also 1,760 × 12,000 = 21,120,000 input tokens, or about 352,000 a minute.

Averages do not cause failures

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%


Vote for new content