Streaming LLM APIs

A person submits a question. The page remains empty for nine seconds. Then the entire answer appears at once.

Streaming can improve the experience if useful output is ready before the full answer. It reduces the wait for initial output, not necessarily total generation time.

Streaming changes the response from one body into a sequence of typed events. That makes the first output visible sooner, and it also creates new contract questions: when is the response complete, how does failure appear after `200