Model Serving

1. The Incident

The store ships its support assistant: a 7B-parameter model, self-hosted for privacy, answering customer questions. Engineering does the obvious: the model on a GPU, requests served one at a time, like every web service ever built. It works. Latency is fine. Then finance runs the numbers.

Do the arithmetic with them. One A100-class GPU: ~4/hour, ~35,000/year: generates about 50 tokens/second serving a single request. An average reply is 400 tokens: 8 seconds of GPU per request, so one GPU sustains ~0.125 requests/second