9.7 Token Rate Limiting for AI APIs

One request to a model endpoint sends 50 tokens and receives 20. The next sends a 100,000-token document and asks for a 4,000-token answer, which is about 1,500 times as many tokens. A classic rate limiter that allows 100 requests per minute per user counts both as one request, even though their cost, latency, and load on the provider's GPUs differ by three orders of magnitude.

Model providers therefore limit tokens per minute (TPM) as well as requests per minute (RPM), and a gateway in front of an AI API should do the same, per tenant, with budgets that protect both capacity and money