Model Serving

1. The Incident

The store deploys a self-hosted 7B-parameter support model for privacy and serves requests one at a time on a GPU, following a conventional web-service design. Response latency meets the requirement, but the resulting capacity calculation reveals a cost problem.

Do the arithmetic with them. One A100-class GPU (about 4/hour, 35,000/year) generates about 50 tokens/second serving a single request. An average reply is 400 tokens. That is 8 seconds of GPU per request, so one GPU sustains about 0