Rate Limiting Expensive Compute

Two customers each make 60 requests in one minute.

The first sends short questions and generates short answers. The second sends large documents and requests thousands of output tokens each time. Both stayed inside a limit of 60 requests per minute. Their compute use was nowhere close.

A request limit protects request-handling overhead. It does not, by itself, protect expensive inference. The API must limit the resource being consumed, reserve for output that does not yet exist, and account for a stream that holds capacity until its terminal event.