Model Serving

1. The Incident

The store ships its support assistant: a 7B-parameter model, self-hosted for privacy, answering customer questions. Engineering does the obvious thing. The model sits on a GPU, and requests are served one at a time, like every web service ever built. It works. Latency is fine. Then finance runs the numbers.

Do the arithmetic with them. One A100-class GPU (about 4/hour, 35,000/year) generates about 50 tokens/second serving a single request. An average reply is 400 tokens