Latency Patterns: Streaming, Parallel Tool Calls, Speculative Work

When a customer waits 44 seconds for Desk to answer, what exactly are they waiting for?

The obvious answer is the model, and it is often wrong. The wait contains model calls, tool calls, and time in which the screen shows nothing, and each has a different repair. Three patterns address them: running independent calls at the same time, showing progress before the work ends, and starting likely work before the model asks for it.

What the Customer Waits For

**Total latency is the time from the customer's message to the end of the reply