Inference Economics

1. Two Meters Are Running

Every GenAI design has two meters running the whole time it serves traffic, and you need to be able to read both of them. A latency meter tracks how fast an answer arrives. You measure it in TTFT (time to first token: how long the user waits before any text shows up) and time per output token, both covered in the previous lesson